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PREFACE 



The subject of numerical analysis has a long history. In fact, it predates by cen- 
turies the existence of the modern computer. Of course, the advent of the modern 
computer in the middle of the twentieth century gave greatly added impetus to the 
subject, and so it now plays a central role in a large part of engineering analysis, 
simulation, and design. This is so true that no engineer can be deemed competent 
without some knowledge and understanding of the subject. Because of the back- 
ground of the author, this book tends to emphasize issues of particular interest to 
electrical and computer engineers, but the subject (and the present book) is certainly 
relevant to engineers from all other branches of engineering. 

Given the importance level of the subject, a great number of books have already 
been written about it, and are now being written. These books span a colossal 
range of approaches, levels of technical difficulty, degree of specialization, breadth 
versus depth, and so on. So, why should this book be added to the already huge, 
and growing list of available books? 

To begin, the present book is intended to be a part of the students' first exposure 
to numerical analysis. As such, it is intended for use mainly in the second year 
of a typical 4-year undergraduate engineering program. However, the book may 
find use in later years of such a program. Generally, the present book arises out of 
the author's objections to educational practice regarding numerical analysis. To be 
more specific 

1. Some books adopt a "grocery list" or "recipes" approach (i.e., "methods" at 
the expense of "analysis") wherein several methods are presented, but with 
little serious discussion of issues such as how they are obtained and their 
relative advantages and disadvantages. In this genre often little consideration 
is given to error analysis, convergence properties, or stability issues. When 
these issues are considered, it is sometimes in a manner that is too superficial 
for contemporary and future needs. 

2. Some books fail to build on what the student is supposed to have learned 
prior to taking a numerical analysis course. For example, it is common for 
engineering students to take a first-year course in matrix/linear algebra. Yet, 
a number of books miss the opportunity to build on this material in a manner 
that would provide a good bridge from first year to more sophisticated uses 
of matrix/linear algebra in later years (e.g., such as would be found in digital 
signal processing or state variable control systems courses). 
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3. Some books miss the opportunity to introduce students to the now quite vital 
area of functional analysis ideas as applied to engineering problem solving. 
Modern numerical analysis relies heavily on concepts such as function spaces, 
orthogonality, norms, metrics, and inner products. Yet these concepts are 
often considered in a very ad hoc way, if indeed they are considered at all. 

4. Some books tie the subject matter of numerical analysis far too closely to 
particular software tools and/or programming languages. But the highly tran- 
sient nature of software tools and programming languages often blinds the 
user to the timeless nature of the underlying principles of analysis. Further- 
more, it is an erroneous belief that one can successfully employ numerical 
methods solely through the use of "canned" software without any knowledge 
or understanding of the technical details of the contents of the can. While 
this does not imply the need to understand a software tool or program down 
to the last line of code, it does rule out the "black box" methodology. 

5. Some books avoid detailed analysis and derivations in the misguided belief 
that this will make the subject more accessible to the student. But this denies 
the student the opportunity to learn an important mode of thinking that is a 
huge aid to practical problem solving. Furthermore, by cutting the student 
off from the language associated with analysis the student is prevented from 
learning those skills needed to read modern engineering literature, and to 
extract from this literature those things that are useful for solving the problem 
at hand. 

The prospective user of the present book will likely notice that it contains material 
that, in the past, was associated mainly with more advanced courses. However, the 
history of numerical computing since the early 1980s or so has made its inclusion 
in an introductory course unavoidable. There is nothing remarkable about this. For 
example, the material of typical undergraduate signals and systems courses was, 
not so long ago, considered to be suitable only for graduate-level courses. Indeed, 
most (if not all) of the contents of any undergraduate program consists of material 
that was once considered far too advanced for undergraduates, provided one goes 
back far enough in time. 

Therefore, with respect to the observations mentioned above, the following is a 
summary of some of the features of the present book: 

1 . An axiomatic approach to function spaces is adopted within the first chapter. 
So the book immediately exposes the student to function space ideas, espe- 
cially with respect to metrics, norms, inner products, and the concept of 
orthogonality in a general setting. All of this is illustrated by several examples, 
and the basic ideas from the first chapter are reinforced by routine use 
throughout the remaining chapters. 

2. The present book is not closely tied to any particular software tool or pro- 
gramming language, although a few MATLAB-oriented examples are pre- 
sented. These may be understood without any understanding of MATLAB 
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(derived from the term matrix laboratory) on the part of the student, how- 
ever. Additionally, a quick introduction to MATLAB is provided in Chapter 
13. These examples are simply intended to illustrate that modern software 
tools implement many of the theories presented in the book, and that the 
numerical characteristics of algorithms implemented with such tools are not 
materially different from algorithm implementations using older software 
technologies (e.g., catastrophic convergence, and ill conditioning, continue 
to be major implementation issues). Algorithms are often presented in a 
Pascal-like pseudocode that is sufficiently transparent and general to allow 
the user to implement the algorithm in the language of their choice. 

3. Detailed proofs and/or derivations are often provided for many key results. 
However, not all theorems or algorithms are proved or derived in detail 
on those occasions where to do so would consume too much space, or not 
provide much insight. Of course, the reader may dispute the present author's 
choices in this matter. But when a proof or derivation is omitted, a reference 
is often cited where the details may be found. 

4. Some modern applications examples are provided to illustrate the conse- 
quences of various mathematical ideas. For example, chaotic cryptography, 
the CORDIC (coordinate rotational digital computing) method, and least 
squares for system identification (in a biomedical application) are considered. 

5. The sense in which series and iterative processes converge is given fairly 
detailed treatment in this book as an understanding of these matters is now 
so crucial in making good choices about which algorithm to use in an appli- 
cation. Thus, for example, the difference between pointwise and uniform 
convergence is considered. Kernel functions are introduced because of their 
importance in error analysis for approximations based on orthogonal series. 
Convergence rate analysis is also presented in the context of root-finding 
algorithms. 

6. Matrix analysis is considered in sufficient depth and breadth to provide an 
adequate introduction to those aspects of the subject particularly relevant to 
modern areas in which it is applied. This would include (but not be limited 
to) numerical methods for electromagnetics, stability of dynamic systems, 
state variable control systems, digital signal processing, and digital commu- 
nications. 

7. The most important general properties of orthogonal polynomials are pre- 
sented. The special cases of Chebyshev, Legendre, and Hermite polynomials 
are considered in detail (i.e., detailed derivations of many basic properties 
are given). 

8. In treating the subject of the numerical solution of ordinary differential 
equations, a few books fail to give adequate examples based on nonlin- 
ear dynamic systems. But many examples in the present book are based on 
nonlinear problems (e.g., the Duffing equation). Furthermore, matrix methods 
are introduced in the stability analysis of both explicit and implicit methods 
for nth-order systems. This is illustrated with second-order examples. 
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Analysis is often embedded in the main body of the text rather than being rele- 
gated to appendixes, or to formalized statements of proof immediately following a 
theorem statement. This is done to discourage attempts by the reader to "skip over 
the math." After all, skipping over the math defeats the purpose of the book. 

Notwithstanding the remarks above, the present book lacks the rigor of a math- 
ematically formal treatment of numerical analysis. For example, Lebesgue measure 
theory is entirely avoided (although it is mentioned in passing). With respect to 
functional analysis, previous authors (e.g., E. Kreyszig, Introductory Functional 
Analysis with Applications) have demonstrated that it is very possible to do this 
while maintaining adequate rigor for engineering purposes, and this approach is 
followed here. 

It is largely left to the judgment of the course instructor about what particular 
portions of the book to cover in a course. Certainly there is more material here 
than can be covered in a single term (or semester). However, it is recommended 
that the first four chapters be covered largely in their entirety (perhaps excepting 
Sections 1.4, 3.6, 3.7, and the part of Section 4.6 regarding SVD). The material of 
these chapters is simply too fundamental to be omitted, and is often drawn on in 
later chapters. 

Finally, some will say that topics such as function spaces, norms and inner 
products, and uniform versus pointwise convergence, are too abstract for engineers. 
Such individuals would do well to ask themselves in what way these ideas are 
more abstract than Boolean algebra, convolution integrals, and Fourier or Laplace 
transforms, all of which are standard fare in present-day electrical and computer 
engineering curricula. 
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Functional Analysis Ideas 



1.1 INTRODUCTION 

Many engineering analysis and design problems are far too complex to be solved 
without the aid of computers. However, the use of computers in problem solving 
has made it increasingly necessary for users to be highly skilled in (practical) 
mathematical analysis. There are a number of reasons for this. A few are as follows. 

For one thing, computers represent data to finite precision. Irrational numbers 
such as it or V2 do not have an exact representation on a digital computer (with the 
possible exception of methods based on symbolic computing). Additionally, when 
arithmetic is performed, errors occur as a result of rounding (e.g., the truncation of 
the product of two n-bit numbers, which might be In bits long, back down to n 
bits). Numbers have a limited dynamic range; we might get overflow or underflow 
in a computation. These are examples of finite-precision arithmetic effects. Beyond 
this, computational methods frequently have sources of error independent of these. 
For example, an infinite series must be truncated if it is to be evaluated on a com- 
puter. The truncation error is something "additional" to errors from finite-precision 
arithmetic effects. In all cases, the sources (and sizes) of error in a computation 
must be known and understood in order to make sensible claims about the accuracy 
of a computer-generated solution to a problem. 

Many methods are "iterative." Accuracy of the result depends on how many 
iterations are performed. It is possible that a given method might be very slow, 
requiring many iterations before achieving acceptable accuracy. This could involve 
much computer runtime. The obvious solution of using a faster computer is usually 
unacceptable. A better approach is to use mathematical analysis to understand why 
a method is slow, and so to devise methods of speeding it up. Thus, an important 
feature of analysis applied to computational methods is that of assessing how 
much in the way of computing resources is needed by a given method. A given 
computational method will make demands on computer memory, operations count 
(the number of arithmetic operations, function evaluations, data transfers, etc.), 
number of bits in a computer word, and so on. 

A given problem almost always has many possible alternative solutions. Other 
than accuracy and computer resource issues, ease of implementation is also rel- 
evant. This is a human labor issue. Some methods may be easier to implement 
on a given set of computing resources than others. This would have an impact 
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on software/hardware development time, and hence on system cost. Again, math- 
ematical analysis is useful in deciding on the relative ease of implementation of 
competing solution methods. 

The subject of numerical computing is truly vast. Methods are required to handle 
an immense range of problems, such as solution of differential equations (ordi- 
nary or partial), integration, solution of equations and systems of equations (linear 
or nonlinear), approximation of functions, and optimization. These problem types 
appear to be radically different from each other. In some sense the differences 
between them are true, but there are means to achieve some unity of approach in 
understanding them. 

The branch of mathematics that (perhaps) gives the greatest amount of unity 
is sometimes called functional analysis. We shall employ ideas from this subject 
throughout. However, our usage of these ideas is not truly rigorous; for example, 
we completely avoid topology, and measure theory. Therefore, we tend to follow 
simplified treatments of the subject such as Kreyszig [1], and then only those ideas 
that are immediately relevant to us. The reader is assumed to be very comfortable 
with elementary linear algebra, and calculus. The reader must also be comfortable 
with complex number arithmetic (see Appendix l.A now for a review if necessary). 
Some knowledge of electric circuit analysis is presumed since this will provide 
a source of applications examples later. (But application examples will also be 
drawn from other sources.) Some knowledge of ordinary differential equations is 
also assumed. 

It is worth noting that an understanding of functional analysis is a tremendous 
aid to understanding other subjects such as quantum physics, probability theory 
and random processes, digital communications system analysis and design, digital 
control systems analysis and design, digital signal processing, fuzzy systems, neural 
networks, computer hardware design, and optimal design of systems. Many of the 
ideas presented in this book are also intended to support these subjects. 



1.2 SOME SETS 

Variables in an engineering problem often take on values from sets of numbers. 
In the present setting, the sets of greatest interest to us are (1) the set of integers 
Z = {. . . -3, -2, -1, 0, 1, 2, 3 . . .}, (2) the set of real numbers R, and (3) the set of 
complex numbers C = [x + jy\j — V— 1, x, y e R}. The set of nonnegative inte- 
gers is Z + = {0, 1, 2, 3, . . . , } (so Z + C Z). Similarly, the set of nonnegative real 
numbers is R + = {x e R\x > 0}. Other kinds of sets of numbers will be introduced 
if and when they are needed. 

If A and B are two sets, their Cartesian product is denoted by A x B — 
{(a, b)\a e A, b e B}. The Cartesian product of n sets denoted Aq, A\, . . . , A n -\ 
is A x A\ x • • • x A„_! = {(a , a u ..., a„-i)\a k e A k }. 

Ideas from matrix/linear algebra are of great importance. We are therefore also 
interested in sets of vectors. Thus, R" shall denote the set of n-element vectors 
with real-valued components, and similarly, C" shall denote the set of n-element 
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SOME SETS 3 

vectors with complex-valued components. By default, we assume any vector x to 
be a column vector: 



(1.1) 



Naturally, row vectors are obtained by transposition. We will generally avoid using 
bars over or under symbols to denote vectors. Whether a quantity is a vector will 
be clear from the context of the discussion. However, bars will be used to denote 
vectors when this cannot be easily avoided. The indexing of vector elements Xk will 
often begin with as indicated in (1.1). Naturally, matrices are also important. Set 
R" xm denotes the set of matrices with n rows and m columns, and the elements are 
real-valued. The notation C" xm should now possess an obvious meaning. Matri- 
ces will be denoted by uppercase symbols, again without bars. If A is an n x m 
matrix, then 

A — [a p , q ] p =o,...,n-l, q=0,...,m-l- (1.2) 

Thus, the element in row p and column q of A is denoted a p , q . Indexing of rows 
and columns again will typically begin at 0. The subscripts on the right bracket "]" 
in (1.2) will often be omitted in the future. We may also write a pq instead of a pq 
where no danger of confusion arises. 

The elements of any vector may be regarded as the elements of a sequence of 
finite length. However, we are also very interested in sequences of infinite length. 
An infinite sequence may be denoted by x — (xk) — (xo, x\, xi, . . .), for which Xk 
could be either real-valued or complex-valued. It is possible for sequences to be 
doubly infinite, for instance, x — (x^) — (..., x-2, x—i, xo, x\, xi, ■ ■ •)• 

Relationships between variables are expressed as mathematical functions, that is, 
mappings between sets. The notation f\A — > B signifies that function / associates 
an element of set A with an element from set B. For example, /|R — > R represents 
a function defined on the real-number line, and this function is also real-valued; 
that is, it maps "points" in R to "points" in R. We are familiar with the idea 
of "plotting" such a function on the xy plane if y = f(x) (i.e., x, y e R). It is 
important to note that we may regard sequences as functions that are defined on 
either the set Z (the case of doubly infinite sequences), or the set Z + (the case 
of singly infinite sequences). To be more specific, if, for example, k e Z + , then 
this number maps to some number x* that is either real-valued or complex-valued. 
Since vectors are associated with sequences of finite length, they, too, may be 
regarded as functions, but defined on a finite subset of the integers. From (1.1) this 
subset might be denoted by Z„ = {0, 1,2, . . . , n — 2,n — 1}. 

Sets of functions are important. This is because in engineering we are often 
interested in mappings between sets of functions. For example, in electric circuits 
voltage and current waveforms (i.e., functions of time) are input to a circuit via volt- 
age and current sources. Voltage drops across circuit elements, or currents through 



TLFeBOOK 



4 FUNCTIONAL ANALYSIS IDEAS 

circuit elements are output functions of time. Thus, any circuit maps functions from 
an input set to functions from some output set. Digital signal processing systems 
do the same thing, except that here the functions are sequences. For example, a 
simple digital signal processing system might accept as input the sequence (x„), 
and produce as output the sequence (y„) according to 

x n + x n+\ ., ,. 

y n = (1.3) 

for which n e Z + . 

Some specific examples of sets of functions are as follows, and more will be 
seen later. The set of real- valued functions defined on the interval [a, b] C R that 
are n times continuously differentiable may be denoted by C n [a, b]. This means 
that all derivatives up to and including order n exist and are continuous. If n — 
we often just write C[a,b], which is the set of continuous functions on the interval 
[a, b]. We remark that the notation [a, b] implies inclusion of the endpoints of the 
interval. Thus, (a, b) implies that the endpoints a and b are not to be included [i.e., 
if x € (a, b), then a < x < b]. 

A polynomial in the indeterminate x of degree n is 

n 

Pn(.x) = ^,Pn,kX k - ( L4 ) 

k=0 

Unless otherwise stated, we will always assume p n \ e R. The indeterminate x 
is often considered to be either a real number or a complex number. But in 
some circumstances the indeterminate x is merely regarded as a "placeholder," 
which means that x is not supposed to take on a value. In a situation like this 
the polynomial coefficients may also be regarded as elements of a vector (e.g., 
p n = [p n o p n j • • • p n , n ] ). This happens in digital signal processing when we 
wish to convolve 1 sequences of finite length, because the multiplication of polyno- 
mials is mathematically equivalent to the operation of sequence convolution. We 
will denote the set of all polynomials of degree n as P" . If x is to be from the 
interval [a, b] C R, then the set of polynomials of degree n on [a, b] is denoted 
by P"[a, b]. If m < n we shall usually assume P m [a, b] C P"[a, b]. 



1.3 SOME SPECIAL MAPPINGS: METRICS, NORMS, 
AND INNER PRODUCTS 

Sets of objects (vectors, sequences, polynomials, functions, etc.) often have cer- 
tain special mappings defined on them that turn these sets into what are commonly 
called function spaces. Loosely speaking, functional analysis is about the properties 

These days it seems that the operation of convolution is first given serious study in introductory signals 
and systems courses. The operation of convolution is fundamental to all forms of signal processing, 
either analog or digital. 
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of function spaces. Generally speaking, numerical computation problems are best 
handled by treating them in association with suitable mappings on well-chosen 
function spaces. For our purposes, the three most important special types of map- 
pings are (1) metrics, (2) norms, and (3) inner products. You are likely to be already 
familiar with special cases of these really very general ideas. 

The vector dot product is an example of an inner product on a vector space, while 
the Euclidean norm (i.e., the square root of the sum of the squares of the elements in 
a real-valued vector) is a norm on a vector space. The Euclidean distance between 
two vectors (given by the Euclidean norm of the difference between the two vectors) 
is a metric on a vector space. Again, loosely speaking, metrics give meaning to the 
concept of "distance" between points in a function space, norms give a meaning 
to the concept of the "size" of a vector, and inner products give meaning to the 
concept of "direction" in a vector space. 2 

In Section 1.1 we expressed interest in the sizes of errors, and so naturally the 
concept of a norm will be of interest. Later we shall see that inner products will 
prove to be useful in devising means of overcoming problems due to certain sources 
of error in a computation. In this section we shall consider various examples of 
function spaces, some of which we will work with later on in the analysis of 
certain computational problems. We shall see that there are many different kinds 
of metric, norm, and inner product. Each kind has its own particular advantages 
and disadvantages as will be discovered as we progress through the book. 

Sometimes a quantity cannot be computed exactly. In this case we may try to 
estimate bounds on the size of the quantity. For example, finding the exact error 
in the truncation of a series may be impossible, but putting a bound on the error 
might be relatively easy. In this respect the concepts of supremum and infimum 
can be important. These are defined as follows. 

Suppose we have E C R. We say that E is bounded above if E has an upper 
bound, that is, if there exists a B e R such that x < B for all x e E. If E ^ 
(empty set; set containing no elements) there is a supremum of E [also called a 
least upper bound (lub)], denoted 

sup E. 

For example, suppose E — [0, 1), then any B > 1 is an upper bound for E, but 
sup E = 1. More generally, sup E < B for every upper bound B of E. Thus, the 
supremum is a "tight" upper bound. Similarly, E may be bounded below. If E has 
a lower bound there is a b e R such that x > b for all x e E. If E ^ 0, then there 
exists an infimum [also called a greatest lower bound (gib)], denoted by 

ME. 

For example, suppose now E = (0, 1]; then any b < is a lower bound for E, 
but inf E — 0. More generally, inf E > b for every lower bound b of E. Thus, the 
infimum is a "tight" lower bound. 

The idea of "direction" is (often) considered with respect to the concept of an orthogonal basis in a 
vector space. To define "orthogonality" requires the concept of an inner product. We shall consider this 
in various ways later on. 
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1.3.1 Metrics and Metric Spaces 

In mathematics an axiomatic approach is often taken in the development of analysis 
methods. This means that we define a set of objects, a set of operations to be 
performed on the set of objects, and rules obeyed by the operations. This is typically 
how mathematical systems are constructed. The reader (hopefully) has already seen 
this approach in the application of Boolean algebra to the analysis and design of 
digital electronic systems (i.e., digital logic). We adopt the same approach here. 
We will begin with the following definition. 

Definition 1.1: Metric Space, Metric A metric space is a set X and a 
function d\X x X — > R + , which is called a metric or distance function on X. 
If x, y, z € X then d satisfies the following axioms: 

(Ml) d(x, y) = if and only if (iff) x = y. 

(M2) d(x, y) — d(y, x) (symmetry property). 

(M3) d(x, y) < d(x, z) + d(z, y) (triangle inequality). 

We emphasize that X by itself cannot be a metric space until we define d. Thus, 
the metric space is often denoted by the pair (X, d). The phrase "if and only 
if probably needs some explanation. In (Ml), if you were told that d(x, y) — 0, 
then you must immediately conclude that x — y. Conversely, if you were told that 
x — y, then you must immediately conclude that d(x, y) — 0. Instead of the words 
"if and only if it is also common to write 

d(x,y) = 0<S-i=)>. 

The phrase "if and only if is associated with elementary logic. This subject is 
reviewed in Appendix l.B. It is recommended that the reader study that appendix 
before continuing with later chapters. 

Some examples of metric spaces now follow. 

Example 1.1 Set X — R, with 

d(x,y) = \x-y\ (1.5) 

forms a metric space. The metric (1.5) is what is commonly meant by the "distance 
between two points on the real number line." The metric (1.5) is quite useful in 
discussing the sizes of errors due to rounding in digital computation. This is because 
there is a norm on R that gives rise to the metric in (1.5) (see Section 1.3.2). 



Example 1.2 The set of vectors R" with 



d(x, y) 



n-\ 



ixk — yk\ 
U=o 



^2 [** ~ y^ 2 



1/2 



(1.6a) 
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forms a (Euclidean) metric space. However, another valid metric on R" is given by 



n-\ 



d\{x,y) = ^\xk-yk\- (1.6b) 

k=Q 

In other words, we can have the metric space (X, d), or (X, d\). These spaces are 
different because their metrics differ. 

Euclidean metrics, and their related norms and inner products, are useful in pos- 
ing and solving least-squares approximation problems. Least-squares approximation 
is a topic we shall consider in detail later. 

Example 1.3 Consider the set of (singly) infinite, complex- valued, and bounded 
sequences 

X — {x — (xq, xi, X2, ■ ■ -)\xk € C, \xk\ < c(x)(all k)}. (l-7a) 

Here c{x) > is a bound that may depend on x, but not on k. This set forms a 
metric space that may be denoted by l°°[0, oo] if we employ the metric 

d(x,y)= sup \x k -yk\- (1.7b) 

keZ+ 

The notation [0, oo] emphasizes that the sequences we are talking about are only 
singly infinite. We would use [— oo, oo] to specify that we are talking about doubly 
infinite sequences. 

Example 1.4 Define i = [a,fi]cR. The set C[a, b] will be a metric space if 

d(x,y) — sup \x(t) - y(t)\. (1.8) 

In Example 1.1 the metric (1.5) gives the "distance" between points on the real- 
number line. In Example 1.4 the "points" are real-valued, continuous functions of 
t € [a, b]. In functional analysis it is essential to get used to the idea that functions 
can be considered as points in a space. 

Example 1.5 The set X in (1.7a), where we now allow c(x) — >• oo (in other 
words, the sequence need not be bounded here), but with the metric 

El \xt — Vi-I 
_L1 — £^_ (i.9) 

/=o 2*+i 1 + |x t - yjtl 

is a metric space. (Sometimes this space is denoted s.) 
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Example 1.6 Let p be a real- valued constant such that p > 1. Consider the 
set of complex-valued sequences 



X = i X = (xo, JCl. *2, 

This set together with the metric 



.)|** € C, £ 1**1' < OO 

k=0 \ 



(1.10a) 



d(x,y) 



X>*-wi" 



.k=Q 



1/P 



(1.10b) 



forms a metric space that we denote by l p [0, oo] 



Example 1.7 Consider the set of complex-valued functions on [a, b] C R 



X 



-HI. 



\x(t)\ dt < oo 



for which 



d(x, y) 



f 



\x(t) - y(t)\ dt 



1/2 



(1.11a) 



(1.11b) 



is a metric. Pair (X, d) forms a metric space that is usually denoted by L [a, b]. 

The metric space of Example 1 .7 (along with certain variations) is very impor- 
tant in the theory of orthogonal polynomials, and in least-squares approximation 
problems. This is because it turns out to be an inner product space too (see 
Section 1.3.3). Orthogonal polynomials have a major role to play in the solution 
of least squares, and other types of approximation problem. 

All of the metrics defined in the examples above may be shown to satisfy the 
axioms of Definition 1.1. Of course, at least in some cases, much effort might be 
required to do this. In this book we largely avoid making this kind of effort. 



1.3.2 Norms and Normed Spaces 

So far our examples of function spaces have been metric spaces (Section 1.3.1). 
Such spaces are not necessarily associated with the concept of a vector space. 
However, normed spaces (i.e., spaces with norms defined on them) are always 
associated with vector spaces. So, before we can define a norm, we need to recall 
the general definition of a vector space. 

The following definition invokes the concept of afield of numbers. This concept 
arises in abstract algebra and number theory [e.g., 2, 3], a subject we wish to avoid 
considering here. 3 It is enough for the reader to know that R and C are fields under 

This avoidance is not to disparage abstract algebra. This subject is a necessary prerequisite to under- 
standing concepts such as fast algorithms for digital signal processing (i.e., fast Fourier transforms, and 
fast convolution algorithms; e.g., see Ref. 4), cryptography and data security, and error control codes 
for digital communications. 
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the usual real and complex arithmetic operations. These are really the only fields 
that we shall work with. We remark, largely in passing, that rational numbers (set 
denoted Q) are also a field under the usual arithmetic operations. 

Definition 1.2: Vector Space A vector space (linear space) over a field K is 
a nonempty set X of elements x, y, z, ■ ■ . called vectors together with two algebraic 
operations. These operations are vector addition, and the multiplication of vectors 
by scalars that are elements of K. The following axioms must be satisfied: 

(VI) If x, y € X, then x + y e X (additive closure). 

(V2) If x, y, z € X, then (x + y) + z — x + (y + z) (associativity). 

(V3) There exists a vector in X denoted (zero vector) such that for all x e X, 

we have x + = + .r — x. 
(V4) For all x e X, there is a vector —x e X such that — x + x — x + 

(—x) — 0. We call — x the negative of a vector. 
(V5) For all x, y e X we have x + y — y + x (commutativity). 
(V6) If x e X and a e K, then the product of a and x is ax, and ax e X. 
(V7) If x, y € X, and a e K, then a(x + y) — ax + ay. 
(V8) If a, b e K, and x e X, then (a + b)x — ax + bx. 
(V9) If a,b e K, and x e X, then ab(x) — a(bx). 
(V10) If x e X, and I e K, then Ix — x multiplication of a vector by a unit 

scalar; all fields contain a unit scalar (i.e., a number called "one"). 



In this definition, as already noted, we generally work only with K — R, or K — C. 
We represent the zero vector by just as we also represent the scalar zero by 0. 
Rarely is there danger of confusion. 

The reader is already familiar with the special instances of this that relate to the 
sets R" and C". These sets are vector spaces under Definition 1.2, where vector 
addition is defined to be 



x + y — 



xo 




yo 




X\ 




y\ 






+ 




— 


_ X n -l 




y n -\ 





xo 
x\ 



VI 



x n -\ + y n -\ 

and multiplication by a field element is defined to be 



oxq 
ax i 



The zero vector is = [00 • • • 00] , and — x — [—xo — x\ ■ ■ ■ — x n -\ 
then the elements of x and y are real-valued, and a e R, but if X ■■ 



(1.12a) 



(1.12b) 



] r .If X = R" 
= C" then the 
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elements of x and y are complex-valued, and a € C. The metric spaces in Exam- 
ple 1.2 are therefore also vector spaces under the operations defined in (1.12a,b). 
Some further examples of vector spaces now follow. 

Example 1.8 Metric space C[a, b] (Example 1.4) is a vector space under the 
operations 

(x + y)(t)=x(t) + y(t), (ax)(t)=ax(t), (1.13) 

where a e R. The zero vector is the function that is identically zero on the interval 
[a,b]. 

Example 1.9 Metric space l 2 [0, oo] (Example 1.6) is a vector space under the 
operations 

x + y — (x ,x\, ...) + (yo,yi, ■ . .) = (xo + yo,xi +yi, ...), 

ax = (axo,axi, . . .). (1-14) 

Here a e C. 

If x, y € / 2 [0, oo], then some effort is required to verify axiom (VI). This 
requires the Minkowski inequality, which is 



OO 


l/P 


J2\x k + yk\ p 


< 


k=0 





x>*l ; 



.k=0 



Up 



Ei»i j 



.k=Q 



!//> 



(1.15) 



Refer back to Example 1.6; here we employ p — 2, but (1.15) is valid for p > 1. 
Proof of (1.15) is somewhat involved, and so is omitted here. The interested reader 
can see Kreyszig [1, pp. 11-15]. 

We remark that the Minkowski inequality can be proved with the aid of the 
Holder inequality 



^\Xkyk\ 



k=Q 



X>*| ; 



.k=0 



Up 



2>*i 



-Jt=0 



1/9 



(1.16) 



for which here n > 1 and — + - = 1. 
' p q 

We are now ready to define a normed space. 



Definition 1.3: Normed Space, Norm A normed space X is a vector space 
with a norm defined on it. If x e X then the norm of x is denoted by 

\\x\\ (read this as "norm of x"). 

The norm must satisfy the following axioms: 

(Nl) ||;c|| > (i.e., the norm is nonnegative). 
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(N2) ||x|| = 0^x = 0. 

(N3) \\ax\\ — | of | ||x||. Here a is a scalar in the field of X (i.e., a e K; see 

Definition 3.2). 
(N4) ||* + y\\ < ||x|| + ||y|| (triangle inequality). 

The normed space is vector space X together with a norm, and so may be properly 
denoted by the pair (X, \\ ■ ||). However, we may simply write X, and say "normed 
space X," so the norm that goes along with X is understood from the context of 
the discussion. 

It is important to note that all normed spaces are also metric spaces, where the 
metric is given by 

d(x,y) = \\x-y\\ (x,yeX). (1.17) 

The metric in (1.17) is called the metric induced by the norm. 

Various other properties of norms may be deduced. One of these is: 

Example 1.10 Prove | ||y|| - ||x|| | < \\y - x\\. 

Proof From (N3) and (N4) 

\\y\\ = \\y-x + x\\<\\y-x\\ + \\x\\,\\x\\ = \\x-y + y\\<\\y-x\\ + \\y\\. 

Combining these, we obtain 

llyll- 11*11 <ILv-*IULvll-ll*ll>-ILv-*ll- 

The claim follows immediately. 

We may regard the norm as a mapping from X to set R: || • |||Z — > R. This 
mapping can be shown to be continuous. However, this requires generalizing the 
concept of continuity that you may know from elementary calculus. Here we define 
continuity as follows. 

Definition 1.4: Continuous Mapping Suppose X — (X, d) and Y — (Y, d) 

are two metric spaces. The mapping T\X — > Y is said to be continuous at a point 
xq e X if for all e > there is a S > such that 

d(Tx, Txo) < e for all x satisfying d(x,xo) < 5. (1.18) 

T is said to be continuous if it is continuous at every point of X. 

Note that Tx is just another way of writing T(x). (R, | • |) is a normed space; that 
is, the set of real numbers with the usual arithmetic operations defined on it is a 
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vector space, and the absolute value of an element of R is the norm of that element. 
If we identify Y in Definition 1.4 with metric space (R, | • |), then (1.18) becomes 

d(Tx, Tx ) = d(\\x\\, 1 1* 1 1) = I 11*11 - Ikoll I < e, d(x,x ) = ||* -* || < 5. 

To make these claims, we are using (1.17). In other words, X and Y are normed 
spaces, and we employ the metrics induced by their respective norms. In addition, 
we identify T with || • ||. Using Example 1.10, we obtain 

I ||*|| -||*oll I < ll*-*oll <<$■ 

Thus, the requirements of Definition 1.4 are met, and so we conclude that norms 
are continuous mappings. 

We now list some other normed spaces. 

Example 1.11 The Euclidean space R" and the unitary space C" are both 
normed spaces, where the norm is defined to be 



■«-i 



I>| 2 



.k=0 



1/2 



(1.19) 



For R" the absolute value bars may be dropped. 4 It is easy to see that d(x, y) — 
||* — y\\ gives the same metric as in (1.6a) for space R". We further remark that 
for n = 1 we have 1 1* 1 1 = |* | . 



Example 1.12 The space l p [0, oo] is a normed space if we define the norm 
to be 

" oo -|1/P 

X><tl" d-20) 

L*=0 



for which d(x, y) — ||* — y\\ coincides with the metric in (1.10b). 

Example 1.13 The sequence space l°°[0, oo] from Example 1.3ofSection 1.3.1 
is a normed space, where the norm is defined to be 



|*|| = sup \x k \, 

k€Z+ 



(1.21) 



and this norm induces the metric of (1.7b). 



4 Suppose z = x + jy (j ■■ 
in general. 



-1, x, y S R) is some arbitrary complex number. Recall that z # \z\ 
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Example 1.14 The space C[a, b] first seen in Example 1.4 is a normed space, 
where the norm is defined by 

||x|| = sup |x(f)|. (1.22) 

Naturally, this norm induces the metric of (1.8). 

Example 1.15 The space L 2 [a, b] of Example 1.7 is a normed space for the 

norm 

-iV2 



11*11 = 



/ 

J a 



\x(t)\ 2 dt 



(1.23) 



This norm induces the metric in (1.11b). 



The normed space of Example 1.15 is important in the following respect. 
Observe that 

||*|| 2 = / \x(t)\ 2 dt. (1.24) 

J a 

Suppose we now consider a resistor with resistance R. If the voltage drop across its 
terminals is v(t) and the current through it is i(t), we know that the instantaneous 
power dissipated in the device is p(t) — v(t)i(t). If we assume that the resistor is 
a linear device, then v(t) = Ri(t) via Ohm's law. Thus 

p(t) = v(t)i(t) = Ri 2 (t). (1.25) 

Consequently, the amount of energy delivered to the resistor over time interval 
t e [a, b] is given by 



J a 



i 2 {t)dt. (1.26) 



If the voltage/current waveforms in our circuit containing R belong to the space 
L 2 [a,b], then clearly E = R\\i\\ 2 . We may therefore regard the square of the L 2 
norm [given by (1.24)] of a signal to be the energy of the signal, provided the 
norm exists. This notion can be helpful in the optimal design of electric circuits 
(e.g., electric filters), and also of optimal electronic circuits. In analogous fashion, 
an element x of space / 2 [0, oo] satisfies 

oo 
IW| 2 = ^kd 2 <oo (1.27) 

[see (1.10a) and Example 1.12]. We may consider \\x\\ 2 to be the energy of the 
single-sided sequence x. This notion is useful in the optimal design of digital filters. 
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1.3.3 Inner Products and Inner Product Spaces 

The concept of an inner product is necessary before one can talk about orthogonal 
bases for vector spaces. Recall from elementary linear algebra that orthogonal 
bases were important in representing vectors. From a computational standpoint, 
as mentioned earlier, orthogonal bases can have a simplifying effect on certain 
types of approximation problem (e.g., least-squares approximations), and represent 
a means of controlling numerical errors due to so-called ill-conditioned problems. 
Following our axiomatic approach, consider the following definition. 

Definition 1.5: Inner Product Space, Inner Product An inner product space 
is a vector space X with an inner product defined on it. The inner product is a 
mapping (•, -)\X x X -> K that satisfies the following axioms: 

(11) (x + y,z) = {x,z) + {y,z). 

(12) (ax, y) — a(x, y). 

(13) (x,y) = {y,x)*. 

(14) (x, x) > 0, and (x, x) = <S> x = 0. 

Naturally, x, y, z € X, and a is a scalar from the field K of vector space X. The 
asterisk superscript on (y, x) in (13) denotes complex conjugation. 5 

If the field of X is not C, then the operation of complex conjugation in (13) is 
redundant. 

All inner product spaces are also normed spaces, and hence are also metric 
spaces. This is because the inner product induces a norm on X 

||x|| = [(;c,*}] 1/2 (1.28) 

for all x e X. Following (1.17), the induced metric is 

d(x, y) = \\x-y\\=[(x-y,x- y)] 1/2 . (1.29) 

Directly from the axioms of Definition 1.5, it is possible to deduce that (for 

x, y, z e X and a,b e K) 

(ax + by,z) =a(x,z) + b(y,z), (1.30a) 

{x,ay) = a*(x,y), (1.30b) 

and 

(x,ay + bz) = a*(x, y) + b*(x, z). (1.30c) 

The reader should prove these as an exercise. 

If z = X + yj is a complex number, then its conjugate is z* = x — yj. 
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We caution the reader that not all normed spaces are inner product spaces. We 
may construct an example with the aid of the following example. 

Example 1.16 Let x, y be from an inner product space. If || • || is the norm 
induced by the inner product, then \\x + y\\ 2 + \\x — y\\ 2 = 2(||x|| 2 + ||y|| 2 ). This 
is the parallelogram equality. 

Proof Via (1.30a,c) we have 



and 



|x + y|| z = (x + y,x + y) = (x, x + y) + (y, x + y) 
= (x,x) + (x,y) + (y,x) + (y,y), 

\x - y\\ 2 - (x -y,x -y) — (x,x - y) - (y,x - y) 
= (x, x) - (x, y) - (y, x) + (y, y). 



Adding these gives the stated result. 

It turns out that the space l p [0, oo] with p ^ 2 is not an inner product space. The 
parallelogram equality can be used to show this. Consider x = (1, 1, 0, 0, . . .), y = 
(1, — 1, 0, 0, . . .), which are certainly elements of l p [0, oo] [see (1.10a)]. We see that 



11*11 = IMI 



,i/p 



,||x + y|| = |l*-yll=2. 



The parallelogram equality is not satisfied, which implies that our norm does not 
come from an inner product. Thus, l p [0, oo] with p ^ 2 cannot be an inner product 
space. 

On the other hand, Z 2 [0, oo] is an inner product space, where the inner product 
is defined to be 



■ y) = Y, x ky*k- 



(1.31) 



i-0 



Does this infinite series converge? Yes, it does. To see this, we need the Cauchy- 
Schwarz inequality. 6 Recall the Holder inequality of (1.16). Let p — 2, so that 
q — 2. Then the Cauchy-Schwarz inequality is 



^2\xkyk\ 



k=Q 



X>*i 2 



.jt=0 



1/2 r 



X>*i 2 



.k=Q 



1/2 



(1.32) 



The inequality we consider here is related to the Schwarz inequality. We will consider the Schwarz 
inequality later on. This inequality is of immense practical value to electrical and computer engineers. 
It is used to derive the matched-filter receiver, which is employed in digital communications systems, 
to derive the uncertainty principle in quantum mechanics and in signal processing, and to derive the 
Cramer-Rao lower bound on the variance of parameter estimators, to name only three applications. 
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Now 



\(x,y)\ = 



J2 Xk yk 

k=0 



< 



£l*«*l- (1-33) 

fc=0 



The inequality in (1.33) follows from the triangle inequality for | • |. (Recall that the 
absolute value operation is a norm on R. It is also a norm on C; if z — x + jy e C, 
then |z| = yx 2 + y 2 .) The right-hand side of (1.32) is finite because x and y are 
in Z 2 [0, oo]. Thus, from (1.33), (x, y) is finite. Thus, the series (1.31) converges. 

It turns out that C[a, b] is not an inner product space, either. But we will not 
demonstrate the truth of this claim here. 

Some further examples of inner product spaces are as follows. 

Example 1.17 The Euclidean space R" is an inner product space, where the 
inner product is defined to be 

71-1 

(x,y) = J^x k y k . (1.34) 

k=Q 

The reader will recognize this as the vector dot product from elementary linear 
algebra; that is, x ■ y = (x, y). It is well worth noting that 

(x,y)=x T y. (1.35) 

Here the superscript T denotes transposition. So, x T is a row vector. The inner 
product in (1.34) certainly induces the norm in (1.19). 

Example 1.18 The unitary space C" is an inner product space for the inner 

product 

«-i 



■?> = !>«£• d-36) 



k=Q 

Again, the norm of (1.19) is induced by inner product (1.36). If H denotes the 
operation of complex conjugation and transposition (this is called Hermitian trans- 
position), then 

y H = [y* yt---y*„- l ] 

(row vector), and 

(x,y) = y H x. (1.37) 

Example 1.19 The space L 2 [a, b] from Example 1.7 is an inner product space 
if the inner product is defined to be 

(x,y) = / x(t)y*(t)dt. (1.38) 
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The norm induced by (1.38) is 



J a 



-1 1/2 



I* 0)1 dt 
This in turn induces the metric in (1.11b). 



(1-39) 



Now we consider the concept of orthogonality in a completely general manner. 

Definition 1.6: Orthogonality Let x, y be vectors from some inner product 
space X. These vectors are orthogonal iff 

<*,y} = 0. 



The orthogonality of x and y is symbolized by writing x 
A, B C X we write x _L A if x _L a for all a e A, and A _ 
and b e B. 



. y. Similarly, for subsets 
B if a _L b for all a € A, 



If we consider the inner product space R 2 , then it is easy to see that 
([10] r , [0 l] r ) = 0, so [01] r , and [10] r are orthogonal vectors. In fact, these 
vectors form an orthogonal basis for R 2 , a concept we will consider more gen- 
erally below. If we define the unit vectors eo — [1 0] T , and e\ — [0 l] 7 , then we 
recall that any x e R 2 can be expressed as x — xo<?o + *iei- (The extension of 
this reasoning to R" for « > 2 should be clear.) Another example of a pair of 
orthogonal vectors would be x — -7=[1 l] r , and y = -7=[1 — l] r • These too form 

an orthogonal basis for the space R 2 . 
Define the functions 



and 



<Kx) 



f{x) 



0, x < and x > 1 

1, 0<x < 1 



0, x < and x > 1 

1. 



0<X < i 



(1.40) 



(1.41) 



1, i <x < 1 



Function <p(x) is called the Haar scaling function, and function i/f(x) is called the 
Haar wavelet [5]. The function 4>(x) is also called an non-return-to-zero (NRZ) 
pulse, and function V0O is a l so called a Manchester pulse [6]. It is easy to con- 
firm that these pulses are elements of L 2 (R) = L 2 (— oo, oo), and that they are 
orthogonal, that is, (</>, xj/) — under the inner product defined in (1.38). This is 
so because 



{^»,^> 



j — • 



4>(x)t//*(x) dx 



Jo 



ij/(x) dx — 0. 
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Thus, we consider cp and xj/ to be elements in the inner product space L 2 (R), for 
which the inner product is 



-/. 



(x,y) = / x(t)y*(t)dt. 

J —00 

It turns out that the Haar wavelet is the simplest example of the more general class 
of Daubechies wavelets. The general theory of these wavelets first appeared in 
Daubechies [7]. Their development has revolutionized signal processing and many 
other areas. 7 The main reason for this is the fact that for any f(t) e L 2 (R) 

oo oo 

/( f )= J2 H (f,*n,k)Mt), (1-42) 

n = -oo k=-oo 

where if n ,k(t) — 2"^ 2 ^r(2 n t — k). This doubly infinite series is called a wavelet 
series expansion for /. The coefficients f n ^ — (f, ifn,k) have finite energy. In 
effect, if we treat either k or n as a constant, then the resulting doubly infinite 
sequence is in the space l 2 [— oo, oo]. In fact, it is also the case that 

oo oo 

J2 J2 \fn,k\ 2 <oo. (1.43) 

n=-oo k=-oo 

It is to be emphasized that the \jr used in (1.42) could be (1.41), or it could be 
chosen from the more general class in Ref. 7. We shall not prove these things in 
this book, as the technical arguments are quite hard. 

The wavelet series is presently not as familiar to the broader electrical and 
computer engineering community as is the Fourier series. A brief summary of the 
Fourier series is as follows. Again, rigorous proofs of many of the following claims 
will be avoided, though good introductory references to Fourier series are Tolstov 
[8] or Kreyszig [9]. If / e L 2 (0, In), then 

OO 

f(t)= J2 f" ei '"> 1 = ^' c 1 - 44 ) 

n =— oo 

where the Fourier (series) coefficients are given by 

/» = T- f* f(t)e-J nt dt. (1.45) 

We may define 

e„(t) = exp(jMf) (t e (0, 2n), n e Z) (1.46) 

For example, in digital communications the problem of designing good signaling pulses for data 
transmission is best treated with respect to wavelet theory. 
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so that we see 

1 C 2n r • i* 

(/,e„> = — j f(t) [e' nt \ dt = f n . (1.47) 

The series (1.44) is the complex Fourier series expansion for /. Note that for 
n, k e Z 

txp[jn(t + 2irk)] = exp[jnt ] exp[2jrjnk] — expljnt]. (1-48) 

Here we have used Euler's identity 

e JX — cosx + j sinx (1-49) 

and cos(2jtIc) — 1, sin(27rfc) = 0. The function e' nt is therefore 2n -periodic; that 
is, its period is 2it. It therefore follows that the series on the right-hand side of 
(3.40) is a 2it -periodic function, too. The result (1.48) implies that, although / in 
(1.44) is initially defined only on (0, 2jt), we are at liberty to "periodically extend" 
/ over the entire real-number line; that is, we can treat / as one period of the 
periodic function 

fit) = J2 f« + 2jt V (1.50) 

keZ 

for which f(t) = f(t) for t e (0, 2n). Thus, series (1.44) is a way to represent 
periodic functions. Because / € L 2 (Q, 2it), it turns out that 

00 

J2 \fn\ 2 <°° (1-51) 

n=— oo 

so that (/„) € l 2 [— oo, oo]. 

Observe that in (1.47) we have "redefined" the inner product on L 2 (0, 2it) to be 

1 f 27T 
(x,y) = — x(t)y*(t)dt (1.52) 

2jt Jo 

which differs from (1.38) in that it has the factor J- in front. This variation also 
happens to be a valid inner product on the vector space defined by the set in (1.1 la). 
Actually, it is a simple example of a weighted inner product. 
Now consider, for n ^ m 

1 f 2n 1 r ■, i2n 

(e„, e m ) = — e jnt e- jmt dt = e J(»-m)t 

2jt Jq 2jrj(n — m) L Jo 

1nj(n-m) _ j 1—1 

0. (1.53) 



2jt j (n — m) 2nj(n — m) 

Similarly 

1 f 27T . . 1 f 27T 

(e„,e„) = — / e^'e-^'dt = — / dt = 1. (1.54) 

lit Jq lit Jo 

So, e n and e m (if n ^ m) are orthogonal with respect to the inner product in (1.52). 
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From basic electric circuit analysis, periodic signals have finite power. Therefore, 
series (1.44) is a way to represent finite power signals. 8 We might therefore consider 
the space L 2 (0, 2jt) to be the "space of finite power signals." From considerations 
involving the wavelet series representation of (1.42), we may consider L 2 (R) to 
be the "space of finite energy signals." Recall also the discussion at the end of 
Section 1.3.2 (last paragraph). 

An example of a Fourier series expansion is the following. 



Example 1.20 Suppose that 
f(t) = 



1, < t < Tt 

-1, Tt < t < 2jt 



(1.55) 



A sketch of this function is one period of a lit -periodic square wave. The Fourier 
coefficients are given by (for n ^ 0) 



1 f 2n 

In — Z / 

2tc Jo 



f(t)e-i'"dt = 



1 

2tc 



rn r2jr 

/ e~ jnt dt - / e- jnt dt 

JO Jtt 



1 

2jt 



jn L Jo jn L Jjt 



1 1 _ e -jnn 

Tt 



jn 



2 
Ttn 



jnn/2 



1 

2tt 

gjnn/l _ e -jnn/2 



-jnjr 



e -jnn + 1 



2./ 



2 
Ttn 



J" 

-jnjz/2 



sin 



(?) 



where we have made use of 



sinx = — [e JX 



(1.56) 



(1.57) 



This is easily derived using the Euler identity in (1.49). For n = 0, it should be 
clear that /o = 0. 

The coefficients /„ in (1.56) involve expressions containing j. Since /(?) is 
real-valued, it therefore follows that we can rewrite the series expansion in such a 
manner as to avoid complex arithmetic. It is almost a standard practice to do this. 
We now demonstrate this process: 



E /.«"" - l - 



oo . -1 1 

J2 -e~^ 2 sin (-«) e'"' + £ V^/ 2 sin (-») e> n 

n=\ ~ n=-oo 

oo -. CO -. 

J2 -e-^l 1 sin (-«) e' nt + ^ -e inn ' 2 sin (-«) e~' r ' 



In fact, using phasor analysis and superposition, you can apply (1.44) to determine the steady-state 
output of a circuit for any periodic input (including, and especially, nonsinusoidal periodic functions). 
This makes the Fourier series very important in electrical/electronic circuit analysis. 
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9 1 

£ J2 - sin (-«) [ e J nt e-'^ 2 + e^e**" 2 ] 

n = \ 

A °° 1 

-E- cos K f -|)] sin (l M ) 



2A1 . /.T 

(1=1 

=1 
Here we have used the fact that (see Appendix l.A) 



e jnt e -jnn/l + e -jnt e Jnn/2 = 2 Rg ^,7,^-^/2] = 2cQS L A _ *\ j _ 

This is so because if z = x + jy, then z + z* — 2x = 2 Re [z]. Since 

cos(a + /J) = cos a cos /J — sin a sin/3, 
we have 

[/ 7T\"| 7TH JIM 

n ( t 1 = cos(nf) cos h sin(nr) sin — . 

However, if n is an even number, then sin(7rn/2) = 0, and if n is an odd number, 
then cos(tt«/2) = 0. Therefore 

A °° 1 

-^-cos[„(r-|)]sin(|„) 

n = \ 

A °° 1 

-^E^ T T sin[(2n + 1)f]sm2 [ (2n + 1) |]' 

but sin 2 [(2n + l)y] = 1, so finally we have 

oo . oo . 

nt)= J2 f" eir "^-J2^—p[ sin ^ 2n + 1 ^- 

n=-oo ,i=0 

It is important to note that the wavelet series and Fourier series expansions have 
something in common, in spite of the fact that they look quite different and indeed 
are associated with quite different function spaces. The common feature is that both 
representations involve the use of orthogonal basis functions. We are now ready to 
consider this in a general manner. 

Begin by recalling from elementary linear algebra that a basis for a vector space 
such an X — R" or X — C" is a set of n vectors, say 

B = {e ,e 1 ,...,e n - l } (1.58) 

such that the elements e^ (basis vectors) are linearly independent. This means that 
no vector in the set can be expressed as a linear combination of any of the others. 
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In general, it is not necessary that (ek, e n ) — for n ^ k. In other words, indepen- 
dence does not require orthogonality. However, if set B is a basis (orthogonal or 
otherwise) then for any x e X (vector space) there exists a set of coefficients from 
the field of the vector space, say, b = {bo, b\ . . . , b n -\}, such that 

n-\ 

x = J^b k e k . (1.59) 

k=Q 

We say that spaces R" and C" are of dimension n. This is a direct reference to the 
number of basis vectors in B. This notion generalizes. 

Now let us consider a sequence space (e.g., / 2 [0, oo]). Suppose x — 
(xq, x\, X2, . . .) € Z 2 [0, oo]. Define the following unit vector sequences: 

e = (1,0,0,0, ...), e\ = (0, 1,0,0, . . .), e 2 = (0,0, 1, 0, . . .), etc. (1.60) 

Clearly 

oo 

X = y^XK*:- (1-61) 

k=Q 

It is equally clear that no vector e k can be expressed as a linear combination of 
any of the others. Thus, the countably infinite set 9 B = {eo, e\, e2, ■ ■ •} forms a 
basis for Z 2 [0, oo]. The sequence space is therefore of infinite dimension because 
B has a countable infinity of members. It is apparent as well that, under the inner 
product defined in (1.31), we have (e n , e m ) — S n - m . Sequence S — (<5„) is called 
the Kronecker delta sequence. It is defined by 

o, «#o ■ (L62) 

Therefore, the vectors in (1.60) are mutually orthogonal as well. So they happen to 
form an orthogonal basis for Z 2 [0, oo]. Of course, this is not the only possible basis. 
In general, given a countably infinite set of vectors {e k \k e Z + } [no longer neces- 
sarily those in (1.60)] that are linearly independent, and such that e k e / 2 [0, oo], 
for any x e / 2 [0, oo] there will exist coefficients a k e C such that 

00 

X = ^, a k e k- (1-63) 

k=Q 

In view of the above, consider the following linearly independent set of vectors 
from some inner product space X: 

B = {e k \e k eX, keZ}. (1.64) 

y A set A is countably infinite if its members can be put into one-to-one (1-1) correspondence with 
the members of the set Z + . This is also equivalent to being able to place the elements of A into 1-1 
correspondence with the elements of Z. 
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Assume that this is a basis for X. In this case for any x e X, there are coefficients 
a k such that 

x — ^2a k e k . (1.65) 

We define the set B to be orthogonal iff for all n , k e Z 

(e n ,e k ) = K-k- (1.66) 

Assume that the elements of B in (1.64) satisfy (1.66). It is then easy to 
see that 

(x,e n ) = (y2a k e k ,e n \ = ^(ate*. e„) (using (II)) 

\ * Ik 

= ^2a k (e k ,e n ) (using (12)) 

= ^<$fc_„aifc (using (1.66)) 

so finally we may say that 

(x,e„) = a n . (1-67) 

In other words, if the basis B is orthogonal, then 

x = *Y]{x, ek)e k . (1.68) 

keZ 

Previous examples (e.g., Fourier series expansion) are merely special cases of this 
general idea. We see that one of the main features of an orthogonal basis is the 
ease with which we can obtain the coefficients a k . Nonorthogonal bases are harder 
to work with in this respect. This is one of the reasons why orthogonal bases are 
so universally popular. 

A few comments on terminology are in order here. Some would say that the 
condition (1.66) on B in (1.64) means that B is an orthonormal set, and we would 
say that condition 

(e n ,e k ) = a n S n - k 

is the condition for B to be an orthogonal set, where a n is not necessarily unity 
(i.e., equal to one) for all n. However, in this book we often insist that orthogonal 
basis vectors be "normalized" so condition (1.66) holds. 

We conclude the present section by considering the following theorem. It was 
mentioned in a footnote that the following Schwarz inequality (or variations of it) 
is of very great value in electrical and computer engineering. 
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Theorem 1.1: Schwarz Inequality Let X be an inner product space, where 
x, y e X. Then 

\{x,y)\ < 11*11 ILvll- (1-69) 

Equality holds iff {x, y] is a linearly dependent set. 

Proof If y — then (x, 0) = 0, and (1.69) clearly holds in this special case. 
Let y ^ 0. For all scalars a in the field of X we must have [via inner product 
axioms and (1.30)] 

< \\x — ay\\ — (x — ay, x — ay) 

= (x, x) - a*(x, y) - a[(y, x) - a*(y, y)]. 

If we select a* — (y, x)/(y, y), then the quantity in the brackets [•] vanishes. Thus 
0< (x,x)-^\{x,y) = ||x|| 2 - l{X,y [ l 

~ (y,y) \\y\\ 2 

[using (x, y) — (y,x)*, i.e., axiom (13)]. Rearranging, this yields 

\(x,y)\ 2 <\\x\\ 2 \\y\\ 2 , 

and the result (1.69) follows (we must take positive square roots as ||x|| > 0, and 
|x|>0). 

Equality holds iff y — 0, or else ||x — ay|| 2 = 0, hence x — ay — [recall 
(N2)], so x = ay, demonstrating linear dependence of x and y. 

We may now see what Theorem 1 . 1 has to say when applied to the special case of 
a vector dot product. 

Example 1.21 Suppose that X is the inner product space of Example 1.17. 
Since 



^2x k y k 

k=0 



\(x,y)\ = 
Yl'l=o x k\ ' we nave from Theorem 1.1 that 



1 1/2 



M-l 




"«-i 


1/2 


"«-l _ 


^2x k y k 


< 


E** 2 




E? 2 


k=0 




-k=Q - 




-k=Q - 



1/2 



(1.70) 



If y k — axk (a e R) for all k e Z n , then 

-l 



E-*^ 



k=Q 



<\E*l 



k=Q 
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and [E£U 2 ] 1/2 =l«l[EZ=S<T, hence 



-n-\ 


1/2 


"n-1 


E** 

-4=0 - 




-k=0 - 



= i«iE- 



jt=0 



Thus, (1.70) does indeed hold with equality when y — ax. 



1.4 THE DISCRETE FOURIER SERIES (DFS) 

The subject of discrete Fourier series (DFS) and its relationship to the complex 
Fourier series expansion of Section 1.3.3 is often deferred to later courses (e.g., 
signals and systems), but will be briefly considered here as an additional example 
of an orthogonal series expansion. 

The complex Fourier series expansion of Section 1.3.3 was for 2jt -periodic 
functions defined on the real-number line. A similar series expansion exists for 
N-periodic sequences such as x — (x„); that is, for N e {2, 3, 4, . . .} C Z, consider 



*>n — / , x n+kN 



(1.71) 



where x — (x„) is such that x n — for n < 0, and forn>N as well. Thus, x is 
just one period of x. We observe that 



Xn+mN 



E x » 



+mN+kN 



E x » 



+ (m+k)N 



/ J X n+r fJ 



(r — m + k). This confirms that x is indeed A'-periodic (i.e., periodic with period 
N). We normally assume in a context such as this that x„ e C. We also regard x 
as a vector: x = [xq x\ ■ ■ ■ xn-\] t e C . An inner product may be defined on 
the space of Af -periodic sequences according to 



(x,y) - (x,y) - y x 



(1.72) 



(recall Example 1.18), where y e C N is one period of y. We assume, of course, 
that x and y are bounded sequences so that (1.72) is well defined. 
Now define e^ — [e^o e k,\ • • • e k,N-\\ T € C N according to 



ek,n = exp 



2n 
i — kn 

N 



(1.73) 



where n e Zpj. The periodization of et — (ek,n) is 



Ck,n — 2_^ e k,n+mN 
meZ 



(1.74) 
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yielding e k = (e k , n ). That (1.73) is periodic with period N with respect to index n 
is easily seen: 



ek,n+mN = exp 



2n 
/ — /c(« + mN) 

N 



exp 



2jt 
AT 



exp [j27r /cm] = ejt.n- 



It can be shown (by exercise) that [using definition (1.72)] 

N-\ 



(e k ,e r ) = (e*, e r ) : 

iV-l 

= J] exp 

M = 



l] eX P 

n=0 



2tt 
- / — rn 



exp 



2tt 
i — kn 

J N 



2jt 
/ — (k — r)n 
J N 



N, k-r = 
0, otherwise 



(1.75) 



Thus, if we consider (e k „), and (e rn ) with k ^ r we find that these sequences are 
orthogonal, and so form an orthogonal basis for the vector space C^. From (1.75) 
we may write 

(e k ,e r ) = NS k - r . (1.76) 

Thus, there must exist another vector X — [Xq X\ ■ ■ ■ Xn-i] t e C^ such that 



N-l 



l v-» 

x„ = — ) j X k exp 



k=Q 



2n 
/' — kn 

N 



(1.77) 



for n e Z«. In fact 



N-l 



\x, e r ) — / t x n£ rn 



«=0 



N-l (N-l 

^E E x * es p 

«=0 I fc=0 

7V-1 fiV-1 



2n 
/ — /cn 



exp 



2tz 
- / — rn 

J N 



= n ^ Xk \ ^ 6XP 

£=0 I n=0 

= -J^X k (NS k - r ) = X r 



2tt 
/ — (A: — r)n 
J N 



k=0 



(1.78) 



That is 



N-l 



Xk=^2*n exp 



n=0 



2jt 
- / — kn 

N 



(1.79) 



for k € Z» 



TLFeBOOK 



THE DISCRETE FOURIER SERIES (DFS) 27 

In (1.77) we see x n+m ^ — x„ for all m e Z. Thus, (x n ) in (1.77) is A^-periodic, 
and so we have x n = -h Z~lk=o ^k exp jjj-kn with X k given by (1.79). Equation 
(1.77) is the discrete Fourier series (DFS) expansion for an Af -periodic complex- 
valued sequence x such as in (1.71). The DFS coefficients are given by (1.79). 
However, it is common practice to consider only x n for n e Z#, which is equivalent 
to only considering the vector x e C N . In this case the vector X e C N given by 
(1.79) is now called the discrete Fourier transform (DFT) of the vector x, and the 
expression in (1.77) is the inverse DFT (IDFT) of the vector X. We observe that 
the DFT, and the IDFT can be concisely expressed in matrix form, where we define 
the DFT matrix 



F = 



cxp l ~i~N kn 



eC NxN , (1.80) 



k,neZ, 



N 



and we see from (1.77) that F ' = j/F* (IDFT matrix). Thus, X — Fx. We remark 
that the symmetry of F (i.e., F — F T ) means that either k or n in (1.80) may be 
interpreted as row or column indices. 

The DFT has a long history, and its invention is now attributed to Gauss 
[10]. The DFT is of central importance to numerical computing generally, but 
has particularly great significance in digital signal processing as it represents a 
numerical approximation to the Fourier transform, and it can also be used to 
efficiently implement digital filtering operations via so-called fast Fourier trans- 
form (FFT) algorithms. The construction of FFT algorithms to efficiently compute 
X — Fx (and x — F~ l X) is rather involved, and not within the scope of the 
present book. Simply note that the direct computation of the matrix- vector product 
X — Fx needs N 2 complex multiplications and N(N — 1) complex additions. For 
N — 2 P (p e {1,2,3, . . .}), which is called the radix-2 case, the algorithm of Coo- 
ley and Tukey [11] reduces the number of operations to something proportional to 
N log 2 N, which is a substantial savings compared to A" 2 operations with the direct 
approach when N is large enough. Essentially, the method in Ref. 1 1 implicitly fac- 
tors F according to F — F p F p -\ ■ ■ ■ F\, where the matrix factors F^ e C NxN are 
sparse (i.e., contain many zero-valued entries). Note that multiplication by zero is 
not implemented in either hardware or software and so does not represent a compu- 
tational cost in the practical implementation of the FFT algorithm. It is noteworthy 
that the algorithm of Ref. 1 1 also has a long history dating back to the work of 
Gauss, as noted by Heideman et al. [10]. It is also important to mention that fast 
algorithms exist for all possible N ^ 2 P [4]. The following example suggests one 
of the important applications of the DFT/DFS. 



Example 1.22 Suppose that x„ — Ae j0n with 9 — jj-m for m = 1,2,..., 
■j — 1 (N is assumed to be even here). From (1.79) using (1.75) 

X k = AN8 m _ k . (1.81) 
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Now suppose instead that x n — Ae ■' , so similarly 



N-\ 



X k - A XI eXP 



«=0 
A7-1 



A J^exp 

n=0 



2tt 
- / — n(m + &) 

7 at 



2tt 
7 — n(./V — m — k) 
J N 



AN8 N - 



(1.82) 



Thus, if now x„ = Acos(0«) = ±A[e^" + e - ^"], then from (1.81) and (1.82), we 
must have 



X£ = j AiV[<$ m _& + 8 N - m -t] 



(1.83) 



We observe that Xt — for all k ^ m, N — m, but that 



X m — \AN, and X w _ m = jAN . 



Thus, X^ is nonzero only for indices k — m and k — N — m corresponding to 
the frequency of (x„), which is 6 — jrtn. The DFT/DFS is therefore quite use- 
ful in detecting "sinusoids" (also sometimes called "tone detection"). This makes 
the DFT/DFS useful in such applications as narrowband radar and sonar signal 
detection. 

Can you explain the necessity (or, at least, the desirability) of the second equality 
inEq. (1.82)? 



APPENDIX 1.A COMPLEX ARITHMETIC 

Here we summarize the most important facts about arithmetic with complex num- 
bers zeC (set of complex numbers). You shall find this material very useful in 
electric circuits, as well as in the present book. 

Complex numbers may be represented in two ways: (1) Cartesian (rectangular) 
form or (2) polar form. First we consider the Cartesian form. 

In this case z € C has the form z — x + jy, where x, y e R (set of real num- 
bers), and j — V— 1- The complex conjugate of z is defined to be z* — x — jy (so 

j* = -;')• 

Suppose that z\ = x\ + jy\ and zi — X2 + jyi are two complex numbers. Addi- 
tion and subtraction are defined as 

z\ ± Z2 — (x\ ± x 2 ) + j(yi ± yz) 
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[e.g., (1 + 2y) + (3 - 5j) = 4 - 3j, and (1 + 2j) - (3 - 5j) = -2 + 7j]. Using 
j 2 — — 1, the product of zi and Z2 is 



Z1Z2 = C*i + 7>i)fe + jyi) 
= x\x 2 + j 2 yiy 2 + jy\. 
= (*i*2 - yiw) + j{x\y 2 + x 2 y\) 



x\x 2 + j 2 y\y2 + jy\X2 + jx\y 2 



We note that 



ZZ* = (x + jy)(x - jy) = x 2 + y 2 = \ Z \ 2 , 



so |z| = *J x 2 + y 2 defines the magnitude of z. For example, (1 + 2j)(3 — 5j) = 
13 + j. The quotient of z\ and zi is defined to be 

ta_ _ zizl _ (x\ + jy\)(x 2 - jy 2 ) 
Zi Ziz\ x 2 + y 2 

_ {x\xi + yin) + j(x 2 y\ - xiy 2 ) 

x 2 + y 2 
_ x\x 2 + y { y 2 x 2 y x - x\y 2 

x 2 + y 2 x 2 + y\ 

where the last equality is z\/z 2 in Cartesian form. 

Now we may consider polar form representations. For z — x + jy, we may 
regard x and y as the x and y coordinates (respectively) of a point in the Cartesian 
plane (sometimes denoted R 2 ). 10 We may therefore express these coordinates in 
polar form; thus, for any x and y we can write 

x — rcos9, y = rsinO, 

where r > 0, and e [0, 2jr), or 6 e (—it, it]. We observe that 

x 2 + y 2 = r 2 (cos 2 9 + sin 2 6) = r 2 , 

so \z\ — r. 

Now recall the following Maclaurin series expansions (considered in greater 
depth in Chapter 3): 



V(-i)"- 1 — 

M=l 
00 



2«-l 



{In- 1)! 

^2n-2 



EX 
(n - 



(2/1-2)! 

H-l 



(B-l)! 

n = l 

This suggests that z may be equivalently represented by the column vector [xy] . The vector inter- 
pretation of complex numbers can be quite useful. 
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These series converge for — oo < x < oo. Observe the following: 



j x _ y^ U x ) _ y^ 

n=\ n=\ 



(jx) 



(2«-l)-l 



(jx) 



(2«-D 



[(2/i-l)-l]! [2/1-1]! 



where we have split the summation into terms involving even n and odd n. Thus, 
continuing 



00 r ;2n-2^2n-2 ;2n-l„2«-l 



*7* 



E 



J"' 'X 



;- *x' 



(2n-2)\ (In- 1)! 



Ei 

«=i 

00 



2n-2 



,2n-2 



„2n-l 



+ y- 



= Et- 1 )"" 



(2/1-2)! •'(2/1- 1)! 

v 2«-2 



n=l 



(2n - 2) 



+iE(-ir 



07 2 " -2 = /"-') 



„2n-l 



n = \ 



(2«- 1)! 



a 2 " -2 = a 2 )" -1 = (-i)"- 1 ) 



= cosx + y sinx. 

Thus, e'* = cosx + j sinx. This is justification for Euler's identity in (1.49). Addi- 
tionally, since e~' x — cosx — j sinx, we have 



,}* -1- *-;* 



= 2 cosx, e 



jx _ a -jx 



= 2j sin x . 



These immediately imply that 



,jx _ „-jx 



■?}X I p—JX 



V 2 

These identities allow for the conversion of expressions involving trig(onometric) 
functions into expressions involving exponentials, and vice versa. The necessity to 
do this is frequent. For this reason, they should be memorized, or else you should 
remember how to derive them "on the spot" when necessary. 
Now observe that 

re 3 — r cos 9 + jr sm6, 

so that if z — x + jy, then, because there exist r and 9 such that x — r cos 6 and 
y — r sin9, we may immediately write 



Z — re J 
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This is z in polar form. For example (assuming that 9 is in radians) 

1 + 7 = v^e^ 74 , -l + y = V2e 37ry/4 , 
1 - j = V^-- 7 '*/ 4 , -1-7 = V2e~ 3n J/ 4 . 

It can sometimes be useful to observe that 

j = e inl2 , -j = e - jn/2 , and - 1 = e ±J7T . 

If z\ = r\e^ 1 , and z 2 = r^e^ 1 , then 

Z lZ2 = ri^W-W - = ^^(ft-fc). 

Z2 r 2 

In other words, multiplication and division of complex numbers is very easy when 
they are expressed in polar form. 

Finally, some terminology. For z — x + jy, we call x the real part of z, and we 
call y the imaginary part of z. The notation is 

x — Re [z], y — Im [z]. 

That is, z = Re [z] + 7 Im [z]. 



APPENDIX 1.B ELEMENTARY LOGIC 

Here we summarize the basic language and ideas associated with elementary logic 
as some of what is found here appears in later sections and chapters of this book. 
The concepts found here appear often in mathematics and engineering literature. 

Consider two mathematical statements represented as P and Q. Each statement 
may be either true or false. Suppose that we know that if P is true, then Q is 
certainly true (allowing the possibility that Q is true even if P is false). Then we 
say that P implies Q, or Q is implied by P, or P is a sufficient condition for Q, 
or symbolically 

P =► Q or Q «= P. 

Suppose that if P is false, then Q is certainly false (allowing the possibility 
that Q may be false even if P is true). Then we say that P is implied by Q, or Q 
implies P, or P is a necessary condition for Q, or 

P «= Q or Q =► P. 

Now suppose that if P is true, then Q is certainly true, and if P is false, then 
Q is certainly false. In other words, P and Q are either both true or both false. 
Then we say that P implies and is implied by Q, or P is a necessary and sufficient 
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condition for Q, or P and Q are logically equivalent, or P if and only if Q, or 
symbolically 

P O Q- 

A common abbreviation for "if and only if is iff. 

The logical contrary of the statement P is called "not P." It is often denoted by 
either P or ~ P. This is the statement that is true if P is false, or false if P is true. 
For example, if P is the statement "x > 1," then ~ P is the statement "x < 1." If 
P is the statement "f(x) ^ for all x e R," then ~ P is the statement "there is 
at least one x e R for which f(x) — 0." We may write 

x 4 - 5x 2 + 4 = «= x — 1 or x — 2, 

but the converse is not true because x 4 — 5x 2 + 4 = is a quartic equation pos- 
sessing four possible solutions. We may write 

x — 3 =>■ x — 3x, 

but we cannot say x 2 — 3x =>■ x — 3 because x = is also possible. 
Finally, we observe that 

P =>■ Q is equivalent to ~ P <^ ~ Q, 
P ■$= Q is equivalent to ~ P =>■ ~ Q, 
P -o- Q is equivalent to ~ P -o- ~ Q; 

that is, taking logical contraries reverses the directions of implication arrows. 
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PROBLEMS 
1.1. (a) Find a, b e R in 



1+27 



a + bj. 



(b) Find r, 9 e R in 



J = r < 



je 



(Of course, choose r > 0, and 9 e (— it, it].) 
1.2. Solve for x e C in the quadratic equation 



x 2 — 2r coaOx + r 2 



0. 



Here r > 0, and 9 e (— jt, jv]. Express your solution in polar form. 

1.3. Let 0, and <p be arbitrary angles (so 0, 4> e R). Show that 

(cos0 + j sin0)(cos0 + j sin</>) = cos(9 + <p) + j sm(9 + <p). 

1.4. Prove the following theorem. Suppose z e C such that 

z — r cos 9 + jr sin6> 



for which r = \z\ > 0, and 9 € (—jt, it]. Let n € {1, 2, 3, . . 
positive integer). The n different nth roots of z are given by 



(i.e., n is a 



.!/« 



COS 



2jtk 



/sin 



9 + 2itk 



for k = 0, 1,2, ...,«- 1. 
1.5. State whether the following are true or false: 

(a) \x\ < 2 =► x < 2 

(b) |*| < 3 «= < x < 3 
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(c) x — y>0=^x>;y>0 

(d) xy — =$■ x — and y — 

(e) x = 10 «= x 2 = lOx 

Explain your answer in all cases. 
1.6. Consider the function 



,. | -x 2 + 2x + 1, < x < 1 

/W-i x 2_ 2x+ 3 1<x<2 



2' 



Find 



sup /(x), inf /(x). 
*e[0,2] *e[0,2] ' 



1.7. Suppose that we have the following polynomials in the indeterminate x: 

n m 

a{x) — 2^aicX , b(x) = } ^ bjX 1 . 
fc=0 ,/=o 



Prove that 



where 



n+m 



:(x) — a(x)b(x) = 2, c l x > 



1=0 



ci — ) y cikbi-k- 

k=Q 

[Comment: This is really asking us to prove that discrete convolution is 
mathematically equivalent to polynomial multiplication. It explains why the 
MATLAB routine for multiplying polynomials is called conv. Discrete con- 
volution is a fundamental operation in digital signal processing, and is an 
instance of something called finite impulse response (FIR) filtering. You will 
find it useful to note that a^ — for k < 0, and k > n, and that b 1 ■■ — for 
j < 0, and j > m. Knowing this allows you to manipulate the summation 
limits to achieve the desired result.] 

1.8. Recall Example 1.5. Suppose that x\ — 2 k+l , and that y^ = 1 for k e Z + . 
Find the sum of the series d(x, y). (Hint: Recall the theory of geometric 
series. For example, ^^ff' = " if a # 1.) 

1.9. Prove that if x ^ 1, then 



<jpl — 7 fCX 



k-\ 

k=\ 
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is given by 

_ l-(«+ l)x" +nx n+l 

(l - x) 1 

What is the formula for S„ when x — 1 ? (Hint: Begin by showing that 

S„ - xS,j = 1 + x + x 2 H h x n ~ l - nx n .) 

1.10. Recall Example 1.1. Prove that d(x, y) in (1.5) satisfies all the axioms for a 
metric. 

1.11. Recall Example 1.18. Prove that (x, y) in (1.36) satisfies all the axioms for 
an inner product. 

1.12. By direct calculation, show that if x, y, z are elements from an inner product 
space, then 

||z-x|| 2 +||z-y|| 2 = i||x-y|| 2 + 2|| Z -I(* + >0|| 2 

(Appolonius' identity). 

1.13. Suppose x, y e R 3 (three-dimensional Euclidean space) such that 

x = [l 1 if, y = [l -1 If. 

Find all vectors z € R 3 such that (x, z) — (y, z) — 0. 

1.14. The complex Fourier series expansion method as described is for / € 
L 2 (0,2ir). Find the complex Fourier series expansion for / € L 2 (0, T), 
where < T < oo (i.e., the interval on which / is defined is now of arbitrary 
length). 

1.15. Consider again the complex Fourier series expansion for / € L 2 (0, lit). 
Specifically, consider Eq. (1.44). If /(f) € R for all t e (0, 2tt), then show 
that /„ = f* n . [The sequence (/„) is conjugate symmetric] Use this to show 
that for suitable a n , b n e R (all n) we have 

00 00 

^ f n e jnt — «o + y^\a n cos(nf) + b„ sin(nf)]- 

n=-oo n=\ 

How are the coefficients a„ and b n related to /„ ? (Be very specific. There 
is a simple formula.) 

1.16. (a) Suppose that / € L 2 (0, 2jt), and that specifically 

r i, o < t < n 

\ h 7T <t <27T ' 

Find /„ in Eq. (1.44) using (1.45); that is, find the complex Fourier series 
expansion for f(t). Make sure that you appropriately simplify your series 
expansion. 
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(b) Show how to use the result in Example 1.20 to find the complex Fourier 
series expansion for f(t) in (a). 

1.17. This problem is about finding the Fourier series expansion for the wave- 
form at the output of a full-wave rectifier circuit. This circuit is used in 
AC/DC (alternating/direct-current) converters. Knowledge of the Fourier 
series expansion gives information to aid in the design of such converters. 



(a) Find the complex Fourier series expansion of 

'2tt 



f(t) = 



sin 



€// 



T\ 



(b) Find the sequences (a n ), and (b n ) in 



v^ T (2nn \ (2nn 

f(t) - a + 2_, a " cos I -f~t I + b n sin I — — 

n=i *- ^ ' ^ 



for f(t) in (a). You need to consider how T is related to T\. 

1.18. Recall the definitions of the Haar scaling function and Haar wavelet in 
Eqs. (1.40) and (1.41), respectively. Define </>*,„(?) = 2 k/2 (p(2 k t - n), and 
f k ,„(t) = 2 k ' 2 ir(2 k t - n). Recall that (/(?), g(t)) = /_°^ f(t)g*(t)dt is the 
inner product for L 2 (R). 

(a) Sketch (j>k, n (t), and Va,n(0- 

(b) Evaluate the integrals 



(c) Prove that 



/oo poo 

4>l„(t)dt, and / f\ n {t)dt. 
-oo J — oo 



(0t,n(O. 0*,m(O) = 5 n- 



1.19. Prove the following version of the Schwarz inequality. For all x, y € X (inner 
product space) 

|Re[(x,y>]|< ||*|| ||y|| 

with equality iff y = fix, and /J € R is a constant. 

[Hint: The proof of this one is not quite like that of Theorem 1.1. Consider 
(ax + y, ax + y) > with a e R. The inner product is to be viewed as a 
quadratic in a.] 

1.20. The following result is associated with the proof of the uncertainty principle 
for analog signals. 

Prove that for f(t) e L 2 (R) such that \t\f(t) € L 2 (R) and / (1) (?) = 
df(t)/dt e L 2 (R), we have the inequality 

2 



Rc 



/ 

y — ( 



tf(t)[f m (t)Tdt 



/OO /•OO 

\tf{t)\ 2 dt / \f m (t)\ 2 dt . 
-oo J L*J — oo 
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1.21. Suppose e k = [et,o e*,i • • • ^,^-2 ^.iv-i] 7 ^ € C^, where 

e*,n = exp 



27T 

/ — /en 

N 



and keZ N . If #, y € C w recall that (x, y) — Y^k=o x Wk- Prove tnat 
(gfc, e r > = N&k-r- Thus, B = {e,t|/c € Z^} is an orthogonal basis for C^. 
Set B is important in digital signal processing because it is used to define 
the discrete Fourier transform. 
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2.1 INTRODUCTION 

In this chapter we consider how numbers are represented on a computer largely with 
respect to the errors that occur when basic arithmetical operations are performed 
on them. We are most interested here in so-called rounding errors (also called 
roundoff errors). Floating-point computation is emphasized. This is due to the 
fact that most numerical computation is performed with floating-point numbers, 
especially when numerical methods are implemented in high-level programming 
languages such as C, Pascal, FORTRAN, and C++. However, an understanding 
of floating-point requires some understanding of fixed-point schemes first, and so 
this case will be considered initially. In addition, fixed-point schemes are used to 
represent integer data (i.e., subsets of Z), and so the fixed-point representation is 
important in its own right. For example, the exponent in a floating-point number 
is an integer. 

The reader is assumed to be familiar with how integers are represented, and 
how they are manipulated with digital hardware from a typical introductory dig- 
ital electronics book or course. However, if this is not so, then some review of 
this topic appears in Appendix 2. A. The reader should study this material now if 
necessary. 

Our main (historical) reference text for the material of this chapter is Wilkin- 
son [1]. However, Golub and Van Loan [4, Section 2.4] is also a good refer- 
ence. Golub and Van Loan [4] base their conventions and results in turn on 
Forsythe et al. [5]. 



2.2 FIXED-POINT REPRESENTATIONS 

We now consider fixed-point fractions. We must do so because the mantissa in a 
floating-point number is a fixed-point fraction. 

We assume that fractions are t + 1 digits long. If the number is in binary, then 
we usually say "t + 1 bits" long instead. Suppose, then, that x is a (t + l)-bit 
fraction. We shall write it in the form 

OO2 = xo.x\X2 ■■■xt-ixt (xk e {0, 1}). (2.1) 
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The notation {x)2 means that x is in base-2 (binary) form. More generally, (x) r 
means that x is expressed as a base-r number (e.g., if r — 10 this would be the 
decimal representation). We use this notation to emphasize which base we are 
working with when necessary (e.g., to avoid ambiguity). We shall assume that 
(2.1) is a two's complement fraction. Thus, bit xq is the sign bit. If this bit is 1, 
we interpret the fraction to be negative; otherwise, it is nonnegative. For example, 
(1.1011)2 = (-0.3125)io- [To take the two's complement of (1.1011)2, first com- 
plement every bit, and then add (0.0001) 2 . This gives (0.0101) 2 = (0.3 125) i .] In 
general, for the case of a (t + l)-bit two's complement fraction, we obtain 

-1<x<1-2"'. (2.2) 

In fact 

(-l)io = (1.00^00)2, (l-2- f )io = (O.U J _ : JT)2. (2.3) 

t bits t bits 

We may regard (2.2) as specifying the dynamic range of the (t + l)-bit two's 
complement fraction representation scheme. Numbers beyond this range are not 
represented. Justification of (2.2) [and (2.3)] would follow the argument for the 
conversion of two's complement integers into decimal integers that is considered 
in Appendix 2.A. 

Consider the set {x e R| — 1 < x < 1 — 2 - '}. In other words, x is a real number 
within the limits imposed by (2.2), but it is not necessarily equal to a (t + l)-bit 
fraction. For example, x — \[2 — 1 is in the range (2.2), but it is an irrational 
number, and so does not possess an exact (t + l)-bit representation. We may choose 
to approximate such a number with t + 1 bits. Denote the (t + l)-bit approximation 
of x as Q[x]. For example, Q[x] might be the approximation to x obtained by 
selecting an element from set 

B = {b n = -1 + 2~'n\n = 0, 1, . . . , 2 t+1 - 1} C R (2.4) 

that is the closest to x, where distance is measured by the metric in Example 1.1. 
Note that each number in B is representable as a (t + l)-bit fraction. In fact, B is the 
entire set of (t + l)-bit two's complement fractions. Formally, our approximation 
is given by 

Q[x] — argmin \x — b„\. (2.5) 

B6{0,l,...,2 t + 1 -l} 

The notation "argmin" means "let Q[x] be the b n for the n in the set 
[0, 1, . . . , 2 t+l — 1} that minimizes \x — b n \." In other words, we choose the 
argument b n that minimizes the distance to x. Some reflection (and perhaps con- 
sidering some simple examples for small t) will lead the reader to conclude that 
the error in this approximation satisfies 

\x-Q[x]\ <2~ (t+1) . (2.6) 
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The error e — x — Q[x] is called quantization error. Equation (2.6) is an upper 
bound on the size (norm) of this error. In fact, in the notation of Chapter 1, if 
||x|| = |;c|, then ||e|| = \\x — Q[x]|| < 2~ ( - t+l \ We remark that our quantization 
method is not unique. There are many other methods, and these will generally lead 
to different bounds. 

When we represent the numbers in a computational problem on a computer, 
we see that errors due to quantization can arise even before we perform any oper- 
ations on the numbers at all. However, errors will also arise in the course of 
performing basic arithmetic operations on the numbers. We consider the sources 
of these now. 

If x, y are coded as in (2.1), then their sum might not be in the range specified 
by (2.2). This can happen only if x and y are either both positive or both negative. 
Such a condition is fixed-point overflow. (A test for overflow in two's complement 
integer addition appears in Appendix 2.A, and it is easy to modify it for the problem 
of overflow testing in the addition effractions.) Similarly, overflow can occur when 
a negative number is subtracted from a positive number, or if a positive number 
is subtracted from a negative number. A test for this case is possible, too, but we 
omit the details. Other than the problem of overflow, no errors can occur in the 
addition or subtraction of fractions. 

With respect to fractions, rounding error arises only when we perform multipli- 
cation and division. We now consider errors in these operations. 

We will deal with multiplication first. Suppose that x and y are represented 
according to (2.1). Suppose also that xq — yo = 0. It is easy to see that the product 
of x and y is given by 



p=*y=(i> 2 ~*)(i> 2 ~ B ) 

\jfc=0 / \n=Q I 



= (x + xi2 l +---+x,2 ')(y + yi2 l + ■ ■ ■ + y,2 ') 

= x y + (xoyi + *iyo)2 _1 H h x t y t 2~ 2 '. (2.7) 

This implies that the product is a (2t + l)-bit number. If we allow x and y to be 
either positive or negative, then the product will also be 2t + 1 bits long. Of course, 
one of these bits is the sign bit. If we had to multiply several numbers together, 
we see that the product wordsize would grow in some proportion to the number of 
factors in the product. The growth is clearly very rapid, and no practical computer 
could sustain this for very long. We are therefore forced in general to round off the 
product p back down to a number that is only t + 1 bits long. Obviously, this will 
introduce an error. 

How should the rounding be done? There is more than one possibility (just as 
there is more than one way to quantize). Wilkinson [1, p. 4] suggests the following. 
Since the product p has the form 

(ph - P0-P1P2--- Pt-iPtPt+i--- P2t (Pk e {0, 1}) (2.8) 
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we may add 2~^ +1 ' to this product, and then simply discard the last t bits of the 
resulting sum (i.e., the bits indexed t + 1 to 2t). For example, suppose t = 4, and 
consider 

0.00111111 = p 
+0.00001000 = 2~ 5 
0.01000111 

Thus, the rounded product is (0.0100)2- The error involved in rounding in this 
manner is not higher in magnitude than j2~ { — 2~ < -' +l \ Define the result of the 
rounding operation to be fx[p] — fx[xy], so then 

\P- fx[p]\< \2~'. (2.9) 

[For the previous example, p — (0.00111111)2, and so fx[p] — (0.0100)2-] It is 
natural to measure the sizes of errors in the same way as we measured the size of 
quantization errors earlier. Thus, (2.9) is an upper bound on the size of the error 
due to rounding a product. As with quantization, other rounding methods would 
generally give other bounds. We remark that Wilkinson's suggestion amounts to 
"ordinary rounding." 

Finally, we consider fixed-point division. Again, suppose that x and y are rep- 
resented as in (2.1), and consider the quotient q — x/y. Obviously, we must avoid 
y — 0. Also, the quotient will not be in the permitted range given by (2.2) unless 
\y\ > \ x \- This implies that when fixed-point division is implemented either the 
dividend x or the divisor y need to be scaled to meet this restriction. Scaling is 
multiplication by a power of 2, and so should be implemented to reduce rounding 
error. We do not consider the specifics of how to achieve this. Another problem 
is that x/y may require an infinite number of bits to represent it. For example, 
suppose 

= (0.0010) 2 = (0.125)io = / 1\ = — 
q (0.0110)2 (0.375)10 Wio ' )2 ' 

The bar over 01 denotes the fact that this pattern repeats indefinitely. Fortunately, 
the same recipe for the rounding of products considered above may also be used 
to round quotients. If fx[q] again denotes the result of applying this procedure to 
q, then 

\q-fx[q]\<\2~'. (2.10) 

We see that the difficulties associated with division in fixed-point representations 
means that fixed-point arithmetic should, if possible, not be used to implement 
algorithms that require division. This forces us to either (1) employ floating-point 
representations or (2) develop algorithms that solve the problem without the need 
for division operations. 

Both strategies are employed in practice. Usually choice 1 is easier. 
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2.3 FLOATING-POINT REPRESENTATIONS 

In the previous section we have seen that fixed-point numbers are of very limited 
dynamic range. This poses a major problem in employing them in engineering 
computations since obviously we desire to work with numbers far beyond the 
range in (2.2). Floating-point representations provide the definitive solution to this 
problem. We remark (in passing) that the basic organization of a floating-point 
arithmetic unit [i.e., digital hardware for floating-point addition and subtraction 
appears in Ref. 2 (see pp. 295-306)]. There is a standard IEEE format for floating- 
point numbers. We do not consider this standard here, but it is summarized in 
Ref. 2 (see pp. 304-306). Some of the technical subtleties associated with the 
IEEE standard are considered by Higham [6]. 

Following Golub and Van Loan [4, p. 61], the set F (subset of R) of floating- 
point numbers consists of numbers of the form 

x — xq.x\X2 ■ ■ ■ x t -\x t x r e , (2.11) 

where xo is a sign bit (which means that we can replace xq by ±; this is done in 
Ref. 4), and r is the base of the representation [typically r — 2 (binary), or r — 10 
(decimal); we will emphasize r — 2]. Therefore, x\ € {0, 1, . . . , r — 2, r — 1} for 
1 < k < t. These are the digits (bits if r = 2) of the mantissa. We therefore see 
that the mantissa is a fraction. l It is important to note that x\ ^ 0, and this has 
implications with regard to how operations are performed and the resulting rounding 
errors. We call e the exponent. This is an integer quantity such that L < e < U. 
For example, we might represent e as an n-bit two's complement integer. We will 
assume this unless otherwise specified in what follows. This would imply that 
(e)2 — e n -\e n -2 ■ ■ ■ e\eo, and so 

-2"" 1 < e < 2" _1 - 1 (2.12) 

(see Appendix A for justification). For nonzero jeF, then 

m < \x\ < M, (2.13a) 

where 

m = r L -\ M = r u (l-r-<). (2.13b) 

Equation (2.13) gives the dynamic range for the floating-point representation. With 
r = 2we see that the total wordsize for the floating-point number is t + n + 1 bits. 
In the absence of rounding errors in a computation, our numbers may initially 
be from the set 

G = {x e R\m < \x\ < M} U {0}. (2.14) 

Including the sign bit the mantissa is (for r = 2) t + 1 bits long. Frequently in what follows we shall 
refer to it as being only t bits long. This is because we are ignoring the sign bit, which is always 
understood to be present. 
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This set is analogous to the set {x e R| — 1 < x < 1 — 2 - '} that we saw in the 
previous section in our study of fixed-point quantization effects. Again following 
Golub and Van Loan [4], we may define a mapping (operator) fl\G — > F. Here 
c — fl[x] (x e G) is obtained by choosing the closest c € F to x. As you might 
expect, distance is measured using 1 1 • 1 1 = | • | , as we did in the previous section. 
Golub and Van Loan call this rounded arithmetic [4], and it coincides with the 
rounding procedure described by Wilkinson [1, pp. 7-11]. 

Suppose that x and y are two floating-point numbers (i.e., elements of F) and 
that "op 7 ' denotes any of the four basic arithmetic operations (addition, subtrac- 
tion, multiplication, or division). Suppose \x op y\ ^ G. This implies that either 
\x op y\ > M (floating-point overflow), or < \x op y\ < m (floating-point under- 
flow) has occurred. Under normal circumstances an arithmetic fault such as over- 
flow will not happen unless an unstable procedure is being performed. The issue 
of "numerical stability" will be considered later. Overflows typically cause runtime 
error messages to appear. The underflow arithmetic fault occurs when a number 
arises that is not zero, but is too small to represent in the set F. This usually poses 
less of a problem than overflow. 2 However, as noted before, we are concerned 
mainly with rounding errors here. If \x op y\ e G, then we assume that the com- 
puter implementation of x op y will be given by fl[x op y]. In other words, the 
operator // models rounding effects in floating-point arithmetic operations. We 
remark that where floating-point arithmetic is concerned, rounding error arises in 
all four arithmetic operations. This contrasts with fixed-point arithmetic wherein 
rounding errors arise only in multiplication and division. 

It turns out that for the floating-point rounding procedure suggested above 

fl[xopy] = (xopy)(l + €), (2.15) 

where 

\e\ < \r l - f {= 2~ { if r = 2). (2.16) 

We shall justify this only for the case r — 2. Our arguments will follow those of 
Wilkinson [1, pp. 7-11]. 

Let us now consider the addition of the base-2 floating-point numbers 

x = xo.xi ■ ■ ■ x t x2 e * (2.17a) 



and 

y = yo-yi ■ ■ ■ y t x2 ey , (2.17b) 



and we assume that |x| > \y\. (If instead \y\ > \x\, then reverse the roles of x and 
y.) If e x — e y > t, then 

fl[x + y] = x. (2.18) 

Underflows are simply set to zero on some machines. 
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For example, if t = 4, and x = 0.1001 x 2 4 , and j = 0.1110 x 2 _1 , then to add 
these numbers, we must shift the bits in the mantissa of one of them so that both 
have the same exponent. If we choose y (usually shifting is performed on the 
smallest number), then y = 0.00000111 x 2 4 . Therefore, x + y = 0.10010111 x 
2 4 , but then fl[x + y] = 0.1001 x2 4 =i. 

Now if instead we have e x — e y < t, we divide y by 2 e *~ e y by shifting its man- 
tissa e x — e y positions to the right. The sum x + 2 e >'~ ex y is then calculated exactly, 
and requires < It bits for its representation. The sum is multiplied by a power of 2, 
using left or right shifts to ensure that the mantissa is properly normalized [recall 
that for x in (2.11) we must have x\ ^ 0]. Of course, the exponent must be modi- 
fied to account for the shift of the bits in the mantissa. The 2f-bit mantissa is then 
rounded off to t bits using //. Because we have \m x \ + 2 e >~ ex \m y \ < 1 + 1 = 2, 
the largest possible right shift is by one bit position. However, a left shift of up to t 
bit positions might be needed because of the cancellation of bits in the summation 
process. Let us consider a few examples. We will assume that t — A. 

Example 2.1 Let x = 0.1001 x 2 4 , and y = 0.1010 x 2 1 . Thus 

0.10010000 x 2 4 

+0.00010100 x 2 4 

0.10100100 x 2 4 

and the sum is rounded to 0.1010 x 2 4 (computed sum). 

Example 2.2 Let x = 0.1111 x 2 4 , and y = 0.1010 x 2 2 . Thus 

0.11110000 x 2 4 

+0.00101000 x 2 4 

1.00011000 x 2 4 

but 1.00011000 x2 4 = 0.100011000 x 2 5 , and this exact sum is rounded to 0.1001 
x 2 5 (computed sum). 

Example 2.3 Let x = 0.1111 x 2" 4 , and y = -.1110 x 2" 4 . Thus 

0.11110000 x 2 -4 

-0.11100000 x 2 -4 



0.00010000 x 2" 4 

but 0.00010000 x 2" 4 = 0.1000 x 2" 7 , and this exact sum is rounded to 0.1000 x 
2 -7 (computed sum). Here there is much cancellation of the bits leading in turn 
to a large shift of the mantissa of the exact sum to the left. Yet, the computed sum 
is exact. 
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We observe that the computed sum is obtained by computing the exact sum, 
normalizing it so that the mantissa sq.s\ ■ ■ ■ s t -is t s t +i • ■ ■ S2t satisfies s\ = 1 (i.e., 
s\ ^ 0), and then we round it to t places (i.e., we apply //). If the normalized 
exact sum is s — m s x 2 es {— x + y), then the rounding error e' is such that \e'\ < 
i2~'2 e '. Essentially, the error e' is due to rounding the mantissa (a fixed-point 
number) according to the method used in Section 2.2. Because of the form of m s , 
\l e ° <\s\< 2 e \ and so 

fl[x + y] = (x + y)(l + e) (2.19) 

which is just a special case of (2.15). This expression requires further explanation, 
however. Observe that 

\s-fl[s]\ _ \s-(s + e')\ _ \e'\ < ^2-'2 g ' 
Us- 1 |*| " |*| - |*| 



which is the relative error 2 due to rounding. Because we have j2 e * < \s\ < 2 e < , 

i2 e %sotl 

3 ~ fl[s] 



this error is biggest when |*| = i2 e % so therefore we conclude that 



< 2 _f . (2.20) 

1*1 

From (2.19) fl[s] = s + se, so that \s - fl[s]\ = \s\\e\, or |e| = \s - fl[s]\/\s\. 
Thus, |e| < 2~' , which is (2.16). In other words, \e'\ is the absolute error, and |e| 
is the relative error. 

Finally, if x — or y — then no rounding error occurs: e — 0. Subtraction 
results do not differ from addition. 

Now consider computing the product of x and y in (2.17). Since x — m x x2 fl , 
and y — m y x 2 e >' with x\ ^ 0, and y\ ^ we must have 

\\ < \m x m y \ < 1. (2.21) 

This implies that it may be necessary to normalize the mantissa of the product with 
a shift to the left, and an appropriate adjustment of the exponent as well. The 2f-bit 
mantissa of the product is rounded to give a r-bit mantissa. If x — 0, or y = (or 
both x and y are zero), then the product is zero. 

In general, if a is the exact value of some quantity and a is some approximation to a, the absolute 
error is \\a — a\\, while the relative error is 

Ik -all 



M 



(a # 0). 



The relative error is usually more meaningful in practice. This is because an error is really "big" or 
"small" only in relation to the size of the quantity being approximated. 
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We may consider a few examples. We will suppose t — 4. Begin with x — 
0.1010 x 2 2 , and y = 0.1111 x 2 1 , so then 

xy = 0.10010110 x 2 3 , 

and so fl[xy] — 0.1001 x 2 3 (computed product). If now x — 0.1000 x 2 4 , y — 
0.1000 x 2 _1 , then, before normalizing the mantissa, we have 

xy = 0.01000000 x 2 3 , 

and after normalization we have 

xy = 0.10000000 x 2 2 

so that fl[xy] = 0.1000 x 2 2 (computed product). Finally, suppose that x — 
0.1010 x 2°, and y = 0.1010 x 2°, so then the unnormalized product is 

xy = 0.01100100 x 2° 

for which the normalized product is 

xy = 0.11001000 x 2"\ 

so finally fl[xy] — 0.1101 x 2 _1 (computed product). 

The application of // to the normalized product will have exactly the same 
effect as it did in the case of addition (or of subtraction). This may be under- 
stood by recognizing that a 2f-bit mantissa will "look the same" to operator // 
regardless of how that mantissa was obtained. It therefore immediately follows 
that 

fl[xy] = (xy)(l + e), (2.22) 

which is another special case of (2.15), and |e| < 2~' , which is (2.16) again. 
Now consider the quotient x/y, for x and y ^ in (2.17), 

x m x x 2 e " m x „ „ 
q = -= „ = — x ?*~ e y =m q x 2 e * (2.23) 

y m y x 1 y m y 

(so m q = m x /m y , and e q — e x — e y ). The arithmetic unit in the machine has an 
accumulator that we assume contains m x and which is "double length" in that it 
is 2t bits long. Specifically, this accumulator initially stores xq.X\ ■ ■ ■ x t • • • 0. If 

( bits 

\m x \ > \m y \ the number in the accumulator is shifted one place to the right, and 
so e q is increased by one (i.e., incremented). The number in the accumulator is 
then divided by m y in such a manner as to give a correctly rounded f-bit result. 
This implies that the computed mantissa of the quotient, say, m q — qo-qi • • -qt, 
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satisfies the normalization condition q\ — 1, so that j < \m q \ < 1. Once again we 

must have 

x 
fl ~ 

.y 



-(l + O (2.24) 

y 



such that |e| < 2~' . Therefore, (2.15) and (2.16) are now justified for all instances 
of op. 

We complete this section with a few examples. Suppose x — 0.1010 x 2 2 , and 
v = 0.1100 x 2" 2 , then 



0.1010 x 2 2 0.10100000 x 2 1 






y 0.1100x2- 2 0.1100x2"' 

0.10100000 A A 

x 2 4 = 0.11010101 x2 4 



0.1100 



so that fl[q] = 0.1101 x 2 4 (computed quotient). Now suppose that x = 0.1110 x 
2 3 , and y = 0.1001 x 2" 2 , and so 



0.1110 x 2 3 0.01110000 x 2 4 






y 0.1001 x 2" 2 0.1001 x 2"- 

0.01110000 fi fi 

x 2 6 = 0.11000111 x2 6 



0.1001 



so that fl[q] = 0.1100 x 2 6 (computed quotient). 

Thus far we have emphasized ordinary rounding, but an alternative implemen- 
tation of // is to use chopping. If x — ± (X^i x k2 ) x 2 e , then, for chopping 
operator //, we have //[jc] = ± (Yl'k=i x k2~ k ) x 2 e (chopping x to ? + 1 bits 
including the sign bit). Thus, the absolute error is 



|e'| = \x - fl[x]\ = J2 x * 2 ~ k U e < 2 e J2 2 ~ k 

\k=t+l / k=t+l 

(as Xk — 1 for all A: > f), but since Yl/k=t+\ 2 ~ k = 2~'> we mus t have 

|e'| = \x- fl[x]\ < 2"'2 e , 

and so the relative error for chopping is 

I* -//[x]| ^2-V =2 _ f+1 
1*1 ^ 

(because we recall that |x| > j2 e ). We see that the error in chopping is somewhat 
bigger than the error in rounding, but chopping is somewhat easier to implement. 
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2.4 ROUNDING EFFECTS IN DOT PRODUCT COMPUTATION 

Suppose x, y e R". We recall from Chapter 1 (and from elementary linear algebra) 
that the vector dot product is given by 

71-1 

(x,y) — x T y — y T x — 'Y^ l x k y k . (2.25) 

k=Q 

This operation occurs in matrix-vector product computation (e.g., y — Ax, where 
A e R" x "), digital filter implementation (i.e., computing discrete-time convolu- 
tion), numerical integration, and other applications. In other words, it is so common 
that it is important to understand how rounding errors can affect the accuracy of a 
computed dot product. 

We may regard dot product computation as a recursive process. Thus 

n— 1 n— 2 

s n -i = ^2x k y k = ^x k y k +x„-\y n -i = s„- 2 + x n -\y n -\. 

k=0 k=0 

So 

sk = sk-i+xkyk (2.26) 

for k — 0, 1, ...,« — 1, and s-\ — 0. Each arithmetic operation in (2.26) is a 
separate floating-point operation and so introduces its own error into the over- 
all calculation. We would like to obtain a general expression for this error. To 
begin, we may model the computation process according to 

s = fl[xoyo] 

s\ = fl[h + fllxiyi]] 

h = fl[s\ + fl[x 2 y2]] 



S n -2 = fltfn-3 + flUn-2y n -2\\ 

Sn-\ = fl[Sn-2 + fl[x n -\y n -\\]- (2.27) 

From (2.15) we may write 

£o = OxoyoHl + 5o) 

Si = [S + (*iyi)(l + «i)](l + ei) 

S 2 = [Si + fey2)(l + 5 2 )](l+e 2 ) 
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S,t-2 = [Sb-3 + fe-23'n-2)(l + 5„_ 2 )](1 + e„-2) 

S B -i = [*n-2 + (x„-iy„-\)(\ + 5„_i)](l + e„_i), (2.28) 

where |5 k | < 2" J (for k = 0, 1, . . . , n - 1), and |e k | < 2 - ' (for fc=l,2, ..., 
n — 1), via (2.16). It is possible to write 4 

n—\ n—\ 

s n -i = ^2x k y k (l + Yk) = Jn-i + y^XkykYk, (2.29) 



fc=0 *:=0 



where 



B-l 

1 + n = (1 + **) ll^ 1 + 6 y)( £ o = 0). (2.30) 

Note that the n notation means, for example 

n 
\\x k — X X\X 2 - --Xn-lXn, (2.31) 

k=0 

where n is the symbol to compute the product of all x k for k = 0, 1, . . . , n. The 
similarity to how we interpret E notation should therefore be clear. 

The absolute value operator is a norm on R, so from the axioms for a norm 
(recall Definition 1.3), we must have 

71-1 

\s n -\ -s„-i\ = \x T y - fl[x T y]\ < ^2\x k y k \\y k \. (2.32) 

k=0 

In particular, obtaining this involves the repeated use of the triangle inequality. 
Equation (2.32) thus represents an upper bound on the absolute error involved in 
computing a vector dot product. Of course, the notation fl[x T y] symbolizes the 
floating-point approximation to the exact quantity x T y. However, the bound in 
(2.32) is incomplete because we need to appropriately bound the numbers yk- 
To obtain the bound we wish involves using the following lemma. 

Lemma 2.1: We have 

l+x<e x , x>0 (2.33a) 

e* < 1 + 1.01*, 0<jc<.01. (2.33b) 

Equation (2.29) is most easily arrived at by considering examples for small n, for instance 

.? 3 = x y (l + «o)(l + e o)(l + ei)(l + e 2 )(l + £3) + *mU + «i)(l + ei)(l + e 2 )(l + £3) 
+ x 2 y 2 (l + « 2 )(1 + e 2 )(l + £3) + *3:V3(1 + <5 3 )(1 + e 3 ), 
and using such examples to "spot the pattern." 
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Proof Begin with consideration of (2.33a). Recall that for — oo < x < oo 



n=0 

Therefore 



e x = T-r- (2-34) 



? x = i+ x + Y2- 



2 



so that 



«=2 

but the terms in the summation are all nonnegative, so (2.33a) follows immediately. 
Now consider (2.33b), which is certainly valid for x — 0. The result will follow 
if we prove 

e x - 1 

< 1.01 (x#0). 

x 

From (2.34) 

> = 1+7 



x ^L(m+ 1)! ^ (m + 1)! 



so we may also equivalently prove instead that 



^ — ^ X 

£—/ (rn 4- 



, (m + 1)! 

m = l 



< o.oi 



for < x < 0.01. Observe that 



x m 111 I 

^2 — = -x + -x 2 + —x 3 H < -x + x 2 + x 3 + x 4 



(m + 1)! 2 6 24 ~ 2 

m — 1 



1 OO -. oo 

-*+5>* = -*+5; 

fc=2 fc=0 



X* - 1 - X 



1 1,1 1+x 

—x — 1 = — X- 



1— x 2 21— x 

It is not hard to verify that 

1 1+x 

-x < 0.01 

2 1 -x _ 

for < x < 0.01. Thus, (2.33b) follows. 
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If n = 1, 2, 3, .... and if < nu < 0.01, then 

(1 + uf < (e")« [via (2.33a)] 

<1 + 1.01mm [via (2.33b)]. (2.35) 

Now if \Si\ < u for i = 0, 1, . . . , n — 1 then 



n( 1 + < 5 <-)<n (1 + l <5 ''l ) - (1 + M) " 



j'=0 i=0 

so via (2.35) 

«-i 

n( 1 + 5 ') ^ 1 + 1-OImm, (2.36) 

where we must emphasize that < nu < 0.01. Certainly there is a <5 such that 

n-\ 

l+S = Y\(l + Si), (2.37) 

i=0 

and so from (2.36), |<5| < 1.01mm. If we identify yfc with S in (2.33) for all k, then 

IXfel < l.Olnw (2.38) 

for which we consider u — 2~' [because in (2.30) both |e,-| and |5,-| < 2~']. Using 
(2.38) in (2.32), we obtain 

M-l 

\x T y - fl[x T y]\ < l.Olnu J2 l*W*l. (2-39) 

k=0 

but YX=o \xkyk\ = E*=o l^llwl, and this may be symbolized as \x\ T \y\ (so that 
\x\ — [|xo||xi| • • • |x„_i|] r ). Thus, we may rewrite (2.39) as 

\x T y - fl[x T y]\ < l.0lnu\x\ T \y\. (2.40) 



Observe that the relative error satisfies 








\x T y 


- fHx T y]\ 

\ X Ty\ 


< 


1.01mm 


l*riyl 

\x T y\ 



(2.41) 

The bound in (2.41) may be quite large if |x| r |;y| 3> | jc ^y | . This suggests the 
possibility of a large relative error. We remark that since u — 2~', nu < 0.01 will 
hold in all practical cases unless n is very large (a typical value for t is t — 56). 

The potentially large relative errors indicated by the analysis we have just made 
are a consequence of the details of how the dot product was calculated. As noted 
on p. 65 of Ref. 4, the use of a double-precision accumulator to compute the dot 
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product can reduce the error dramatically. Essentially, if x and y are floating- 
point vectors with f-bit mantissas, the "running sum" s k [of (2.26)] is built up in 
an accumulator with a 2f-bit mantissa. Multiplication of two f-bit numbers can 
be stored exactly in a double-precision variable. The large dynamic floating-point 
range limits the likelihood of overflow/underflow. Only when final sum s n -\ is 
written to a single-precision memory location will there be a rounding error. It 
therefore follows that when this alternative procedure is employed, we get 

fl[x T y] = x T y(l + S) (2.42) 

for which \S\ « 2~' (= u). Clearly, this is a big improvement. 
The material of this section shows 

1. The analysis required to obtain insightful bounds on errors can be quite 
arduous. 

2. Proper numerical technique can have a dramatic effect in reducing errors. 

3. Proper technique can be revealed by analysis. 

The following example illustrates how the bound on rounding error in dot prod- 
uct computation may be employed. 

Example 2.4 Assume the existence of a square root function such that 
fl\_\fx\ = V*(l + € ) an d l e l < u - We use the algorithm that corresponds to the 
bound of Eq. (2.40) to compute x T x (x e R"), and then use this to give an algo- 
rithm for ||x|| = V ' x 1 x. This can be expressed in the form of pseudocode: 

s_i := 0; 

for k := to n - 1 do begin 

$ k :=S k _i +x2; 

end; 
||x 1 1 := VSnZT; 

We will now obtain a bound on the relative error due to rounding in the computation 
of | \x 1 1 . We will use the fact that Vl +x < 1 + x (for x > 0). 



Now 



€\ — = =>■ fl[x x] — x x(l + ei), 



and via (2.41) 



\e\\ < l.Olnu 



M T \x\ 
\x T x\ 



\\x\\ 2 
1.01mm r- = I.OIhm 

\\x\\ 2 

-n-l |„. |2 _ V-n-1 2 _ i|^||2 „j IIUII2 



(M 1 |*| = Ejto M -T." k Zo4 = \M\\ and |||*|| 2 | = ||*|| 2 ). So in "short- 
hand" notation, fl[y/ fl[x T x]] = fl[\\x\\], and 



fl[\\x\\] = Vx T xJl + ei (l + e) = ||*||Vl + ei(l + e), 
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and VI + e i < 1 + €1, so 

//[||*l|]< 11*11(1 +ei)(l + e). 
Now (1 + ei)(l + e) — 1 + €\ + € + e\e, implying that 

||jc||(1 + ei)(l + e) = 11*11 + ||*||(6i +€ + e x e) 
so therefore 

//[||*l|]< 11*11 + ll*ll(ei + e + eie), 
and thus 

//[||*l|]- 11*11 



11*11 



< \e\ + 6 + eie| < u + 1.01mm + l.Olnu 2 



= u[l + 1.01/1+ l.Olrew]. 
Of course, we have used the fact that |e| < u. 

2.5 MACHINE EPSILON 

In Section 2.3 upper bounds on the error involved in applying the operator // were 
derived. Specifically, we found that the relative error satisfies 

I* - fl[x]\ J 2~' (rounding) 

— I n-t+\ /„i „; \ ■ \^-^J) 



|*| " \ 2 t+1 (chopping) 

As suggested in Section 2.4, these bounds are often denoted by u; that is, u — 2~' 
for rounding, and u — 2~ t+i for chopping. The bound u is often called the unit 
roundoff [4, Section 2.4.2]. 

The details of how floating-point arithmetic is implemented on any given com- 
puting machine may not be known or readily determined by the user. Thus, u 
may not be known. However, an "experimental" approach is possible. One may 
run a simple program to "estimate" u, and the estimate is the machine epsilon, 
denoted e^- The machine epsilon is defined to be the difference between 1.0 and 
the next biggest floating-point number [6, Section 2.1]. Consequently, cm = 2~ t+1 . 
A pseudocode to compute €m is as follows: 

stop := 1; 

eps := 1.0; 

while stop == 1 do begin 

eps := eps/2.0; 

x := 1.0 + eps; 

ifx< 1.0 
begin 
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stop := 0; 
end; 
end; 
eps := 2.0 * eps; 

This code may be readily implemented as a MATLAB routine. MATLAB stores 
eps (= €m) as a built-in constant, and the reader may wish to test the code above 
to see if the result agrees with MATLAB eps (as a programming exercise). 

In this book we shall (unless otherwise stated) regard machine epsilon and unit 
roundoff as practically interchangeable. 



APPENDIX 2A REVIEW OF BINARY NUMBER CODES 

This appendix summarizes typical methods used to represent integers in binary. 
Extension of the results in this appendix to fractions is certainly possible. This 
material is normally to be found in introductory digital electronics books. The 
reader is here assumed to know Boolean algebra. This implies that the reader 
knows that + can represent either algebraic addition, or the logical or operation. 
Similarly, xy might mean the logical and of the Boolean variables x and y, or it 
might mean the arithmetic product of the real variables x and y. The context must 
be considered to ascertain which meaning applies. 

Below we speak of "complements." These are used to represent negative inte- 
gers, and also to facilitate arithmetic with integers. We remark that the results of 
this appendix are presented in a fairly general manner. Thus, the reader may wish, 
for instance, to see numerical examples of arithmetic using two's complement (2's 
comp.) codings. The reader can consult pp. 276-280 of Ref. 2 for such examples. 
Almost any other books on digital logic will also provide a source of numerical 
examples [3], 

We may typically interpret a bit pattern in one of four ways, assuming that the 
bit pattern is to represent a number (negative or nonnegative integer). An example 
of this is as follows, and it provides a summary of common representations (e.g., 
for n = 3 bits): 



Bit Pattern 


Unsi 


gned Integer 


2's Comp. 


l's Comp. 


Sign 


Magnitude 



























1 




1 


1 


1 




1 


1 







2 


2 


2 




2 


1 


1 




3 


3 


3 




3 


1 







4 


-4 


-3 




-0 


1 


1 




5 


-3 


-2 




-1 


1 1 







6 


-2 


-1 




-2 


1 1 


1 




7 


-1 


-0 




-3 
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In the four coding schemes summarized in this table, the interpretation of the bit 
pattern is always the same when the most significant bit (MSB) is zero. A similar 
table for n — 4 appears in Hamacher et al. [2, see p. 271]. 

Note that, philosophically speaking, the table above implies that a bit pattern 
can have more than one meaning. It is up to the engineer to decide what meaning 
it should have. Of course, this will be a function of purpose. Presently, our purpose 
is that bit patterns should have meaning with respect to the problems of numerical 
computing; that is, bit patterns must represent numerical information. 

The relative merits of the three signed number coding schemes illustrated in the 
table above may be summarized as follows: 



Coding Scheme 



Advantages 



Disadvantages 



2's complement 
l's complement 

Sign magnitude 



Simple adder/subtracter 

circuit 
Only one code for 
Easy to obtain the l's comp. 

of a number 



Intuitively obvious code 



Circuit for finding the 2's comp. 

more complex than circuit for 

finding the l's comp. 
Circuit for addition and 

subtraction more complex 

than for the 2's comp. 

adder/subtracter 
Two codes for 
Has the most complex 

adder/subtracter circuit 
Two codes for 



The following is a summary of some formulas associated with arithmetic (i.e., 
addition and subtraction) with r's and (r — l)'s complements. In binary arithmetic 
r — 2, while in decimal arithmetic r — 10. We emphasize the case r — 2. 

Let A be an n -digit base-r number (integer) 



in-i' 



-2 ' ' 



AiA 



l^o 



where Ak e {0, 1, . . . , r — 2, r — 1}. Digit A n -\ is the most significant digit 
(MSD), while digit A Q is the least significant digit (LSD). Provided that A is 
not negative (i.e., is unsigned), we recognize that to convert A to a base-10 repre- 
sentation (i.e., ordinary decimal number) requires us to compute 



I>A 



k=0 



If A is allowed to be a negative integer, the usage of this summation needs modi- 
fication. This is considered below. 

The r's complement of A is defined to be 



r's complement of A — A* — 



A, A#0 
A = 



(2.A.1) 
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The (r — l)'s complement of A is defined to be 

(r - l)'s complement of A = A = (r n - 1) - A (2.A.2) 

It is important not to confuse the bar over the A in (2. A. 2) with the Boolean not 
operation, although for the special case of r — 2 the bar will denote complemen- 
tation of each bit of A; that is, for r — 2 



^b-i-Ab-2 • • • AjAq 



where the bar now denotes the logical not operation. More generally, if A is a 
base-r number 

A = (r-l)-A n -i (r-l)-A„_ 2 • • • (r - 1) - Aj (r - 1) - A 

Thus, to obtain A, each digit of A is subtracted from r — 1. As a consequence, 
comparing (2A.1) and (2 A. 2), we see that 

A* = A+1 (2A.3) 

where the plus denotes algebraic addition (which takes place in base r). 

Now we consider the three (previously noted) different methods for coding 
integers when r — 2: 

1. Sign-magnitude coding 

2. One's complement coding 

3. Two's complement coding 

In all three of these coding schemes the most significant bit (MSB) is the sign bit. 
Specifically , if A„_i = 0, the number is nonnegative, and if A„_i = 1, the number 
is negative. It can be shown that when the complement (either one's or two's) of 
a binary number is taken, this is equivalent to placing a minus sign in front of the 
number. As a consequence, when given a binary number A = A„_i A„_2 • • • A\ A$ 
coded according to one of these three schemes, we may convert that number to a 
base- 10 integer according to the following formulas: 

1. Sign-Magnitude Coding. The sign-magnitude binary number A = A n -\A n -2 
■ ■ ■ Ai Aq (A* € {0, 1}) has the base-10 equivalent 

n-2 



£Aj2'', A„_i=0 
i=0 

n-2 

-^A,-2', A„_! = l 



(2.A.4) 
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With this coding scheme there are two codings for zero: 
(0)i = (000 • • • 00) 2 = (100 • • • 00) 2 



57 



2. One's Complement Coding. In this coding we represent —A as A. The one's 
complement binary number A = A„_i A„_2 • • • A\ Aq (A# € {0, 1}) has the 
base- 10 equivalent 



n-2 



I>2< 


,An-l=0 


n-2 


, A„_i = 1 



(2.A.5) 



With this coding scheme there are also two codes for zero: 
(0)io = (000- -00)2 = (HI •••11)2 



3. Two's Complement Coding. In this coding we represent —A as A* (— A + 1). 
The two's complement binary number A = A„_i A„_2 • • • A\ Ao (At € {0, 1}) 
has the base- 10 equivalent 



-TT^An-i+^Aa 



(2.A.6) 



The proof is as follows. If A n -\ = 0, then A > and immediately the base- 10 
equivalent is A = Yl"i=o A 2' (via the procedure for converting a number in 
base-2 to one in base- 10), which is (2. A. 6) for A„_i = 0. Now, if A„_i = 1, 
then A < 0, and so if we take the two's complement of A we must get |A|: 

\A\ = A+l 

= (1 - A„_i)(l - A„_ 2 ) • • • (1 - Ai)(l - A ) + 00 •• • 01 



n-\ 



= ^(1 - A,)2' + 1 

i=0 

= 2"- 1 (l-A„_ 1 ) + ^(l-A i )2 i + l 

i=0 

ii—2 M—2 

= ^2'' + l-^A,-2 i '(A„_ 1 = l) 



n-2 



i=0 



i=0 
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1 _ 2"" 1 ^ ,. / " .. 1 - a n+l ' 



'-!>* "■Z>'-T=r 



1-2 

i=0 \ (=0 

n-2 



= 2 »-i _ J^ A,-2'' 



i=0 



and so A = -2"" 1 + £"r o 2 A,- 2'', which is (2.A.6) for A„_i = 1. 
In this coding scheme there is only one code for zero: 



(0)io = (000- • -00)2 



When n-bit integers are added together, there is the possibility that the sum may 
not fit in n bits. This is overflow. The condition is easy to detect by monitoring 
the signs of the operands and the sum. Suppose that x and y are n-bit two's 
complement coded integers, so that the sign bits of these operands are x n -\ and 
y n -\. Suppose that the sum is denoted by s, implying that the sign bit is s n -\. The 
Boolean function that tests for overflow of s — x + y (algebraic sum of x and y) is 

T — x n -\y n -\l n -\ +x n -{y n _ l s n -\. 

The first term will be logical 1 if the operands are negative while the sum is 
positive. The second term will be logical 1 if the operands are positive but the sum 
is negative. Either condition yields T = 1, thus indicating an overflow. A similar 
test may be obtained for subtraction, but we omit this here. 

The following is both the procedure and the justification of the procedure for 
adding two's complement coded integers. 

Theorem 2.A.1: Two's Complement Addition If A and B are n-bit two's 
complement coded numbers, then compute A + B (the sum of A and B) as though 
they were unsigned numbers, discarding any carryout. 

Proof Suppose that A > 0, B > 0; then A + B will generate no carryout from 
the bit position n — \ since A„_i = B n -\ — (i.e., the sign bits are zero-valued), 
and the result will be correct if A + B < 2 n ~ l . (If this inequality is not satisfied, 
then the sign bit will be one, indicating a negative answer, which is wrong. This 
amounts to an overflow.) 

Suppose that A > B > 0; then 

A + (-B) = A + B* = A + 2" - B = 2" + A - B, 

and if we discard the carryout, this is equivalent to subtracting 2" (because the 
carryout has a weight of 2"). Doing this yields A + (— B) — A — B. 
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Similarly 

(-A) + B = A* + B = 2 n - A + B = 2" + B - A, 

and discarding the carry out yields (—A) + B — B — A. 
Again, suppose that A > B > 0, then 

(-A) + (-B) = A* + B* =2 n - A + 2 n - B = 2" + [2 n -(A + B)] 
= 2" + (A + B)* 

so discarding the carryout gives (—A) + (— B) = (A + £)*, which is the desired 
two's complement representation of —(A + B), provided A + B < 2" _1 . (If this 
latter inequality is not satisfied, then we have an overflow.) 

The procedure for subtraction (and its justification) follows similarly. We omit 
these details. 
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PROBLEMS 

2.1. Let /x[x] denote the operation of reducing x (a fixed-point binary fraction) 
to t + 1 bits (including the sign bit) according to the Wilkinson rounding 
(ordinary rounding) procedure in Section 2.2. Suppose that a — (0.1000)2, 
b — (0.1001)2, and c = (0.0101)2, so t — 4 here. In arithmetic of unlimited 
precision, we always have aib + c) — ab + ac. Suppose that a practical com- 
puting machine applies the operator fx[-] after every arithmetic operation. 

(a) Find x — fx[fx[ab] + fx[ac]]. 

(b) Find v = fx[afx[b + c]]. 

Do you obtain x — yl 
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This problem shows that the order of operations in an algorithm implemented 
on a practical computer can affect the answer obtained. 

2.2. Recall from Section 2.2 that 

Find the absolute error in representing q as a (t + l)-bit binary number. Find 
the relative error. Assume both ordinary rounding and chopping (defined at 
the end of Section 2.3 with respect to floating-point arithmetic). 

2.3. Recall that we define a floating-point number in base r to have the form 

x = xq.X\X2 ■ ■ ■ x t -\x t x r e , 



=/ 

where xq € {+, — } (sign digit), Xk £ {0, 1, . . . , r — 1} for k = 1, 2, . . . , t, e 
is the exponent (a signed integer), and x\ ^ (so r~ l < \f\ < 1) if x ^ 0. 
Show that for x ^ 

m <\x\ < M, 

where for L < e < U, we have 



m 



- L ~\ M = r u (l-r-'). 



2.4. Suppose r — 10. We may consider the result of a decimal arithmetic operation 
in the floating-point representation to be 



= ±\Yx k io- k \ xlf/. 



(a) If fl[x] is the operator for chopping, then 

fl[x] — (±.xiX2 • • -x t -\x t ) x 10 e , 

thus, all digits x& for k > t are forced to zero. 

(b) If fl[x] is the operator for rounding then it is defined as follows. Add 
0.00 ■ ■ -01 to the mantissa if x t+ \ > 5, but if x t+ \ < 5, the mantissa is 

t+\ digits 

unchanged. Then all digits Xk for k > t are forced to zero. 

Show that the absolute error for chopping satisfies the upper bound 

\x- fl[x]\ < 10 _f 10 e , 
and that the absolute error for rounding satisfies the upper bound 

\x- fl[x]\ < ilCT r 10 e . 
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Show that the relative errors satisfy 

\x-fl[x]\ \ 10 1 "' (chopping) 



5IO 1 ' (rounding) 

2.5. Suppose that t — 4 and r — 2 (i.e., we are working with floating-point binary 
numbers). Suppose that we have the operands 

x = 0.1011 x 10" 3 , y = -0.1101 x 10 2 . 

Find x + y, x — y, and xy. Clearly show the steps involved. 

2.6. Suppose that A e R" x " ; x e R", and that fl[Ax] represents the result 
of computing the product Ax on a floating-point computer. Define \A\ — 
[|«ijlL-,y=0,i,...,n-i. and \x\ = [|*oll*il"- \x n -i\] T - We have 

fl[Ax] — Ax + e, 

where e e R" is the error vector. Of course, e models the rounding errors 
involved in the actual computation of product Ax on the computer. Justify 
the bound 

\e\ < l.01nu\A\\x\. 

2.7. Explain why a conditional test such as 

if x /y then begin 
f:=f/(x-y); 
end; 

is unreliable. 

(Hint: Think about dynamic range limitations in floating-point arithmetic.) 

2.8. Suppose that x — [xqX\ ■ ■ ■ x n -{\ T is a real-valued vector, ||x||o, 

maxo<A:<n-i |jcjt|, and that we wish to compute ||x||2 = S^=o x k\ 
Explain the advantages, and disadvantages of the following algorithm with 
respect to computational efficiency (number of arithmetic operations, and 
comparisons), and dynamic range limitations in floating-point arithmetic: 

m := llxlloo; 

s:=0; 

for k := to n - 1 do begin 

s :=s+ (x k /m) 2 ; 

end; 
||x|| 2 :=mVs; 

Comments regarding computational efficiency may be made with respect to 
the pseudocode algorithm in Example 2.4. 



1/2 
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2.9. Recall that for x + bx + c = 0, the roots are 



x\ 



-b + \/b 2 - Ac 



X2 



-b - *Jb 2 - Ac 



If b = —0.3001, c = 0.00006, then the "exact" roots for this set of parame- 
ters are 

xi = 0.29989993, x 2 = 2.0006673 x 10" 4 . 

Let us compute the roots using four-digit (i.e., t = 4) decimal (i.e., r = 10) 
floating-point arithmetic, where, as a result of rounding quantization b, and 
c are replaced with their approximations 

b = -0.3001 =b, c = 0.0001 #c. 

Compute J2, which is the approximation to x 2 obtained using b and c in 
place of b and c. Show that the relative error is 



X2 — X2 



X2 



0.75 



(i.e., the relative error is about 75%). {Comment: This is an example of 
catastrophic cancellation.) 

2.10. Suppose a, b e R, and x — a — b. Floating-point approximations to a and b 
are a — fl[a] — a{\ + e a ) and b — fl[b] — b(\ + €b), respectively. Hence 
the floating-point approximation to x is x — a — b. Show that the relative 
error is of the form 



e = 



< a- 



\a\ + \b\ 



What is a? When is |e| large? 
2.11. For a ^ 0, the quadratic equation ax 2 + bx + c — has roots given by 

—b + Vb 2 - Aac -b - \Jb 2 - Aac 



M 



2a 



x 2 



2a 



For c ^ 0, quadratic equation ex + bx + a — has roots given by 



-b + sjb 2 - Aac 
1c 



-b — *Jb 2 — Aac 
Yc ' 



(a) Show that x\x' 2 — 1 and X2x[ — 1. 

(b) Using the result from Problem 2.10, explain accuracy problems that can 
arise in computing either x\ or X2 when b 2 ^> \Aac\. Can you use the 
result in part (a) to alleviate the problem? Explain. 
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3.1 INTRODUCTION 

Sequences and series have a major role to play in computational methods. In this 
chapter we consider various types of sequences and series, especially with respect 
to their convergence behavior. A series might converge "mathematically," and yet 
it might not converge "numerically" (i.e., when implemented on a computer). Some 
of the causes of difficulties such as this will be considered here, along with possible 
remedies. 



3.2 CAUCHY SEQUENCES AND COMPLETE SPACES 

It was noted in the introduction to Chapter 1 that many computational processes 
are "iterative" (the Newton-Raphson method for finding the roots of an equation, 
iterative methods for linear system solution, etc.). The practical effect of this is to 
produce sequences of elements from function spaces. The sequence produced by the 
iterative computation is only useful if it converges. We must therefore investigate 
what this means. 

In Chapter 1 it was possible for sequences to be either singly or doubly infinite. 
Here we shall assume sequences are singly infinite unless specifically stated to the 
contrary. 

We begin with the following (standard) definition taken from Kreyszig 
[1, pp. 25-26]. Examples of applications of the definitions to follow will be con- 
sidered later. 

Definition 3.1: Convergence of a Sequence, Limit A sequence (x n ) in a 
metric space X — (X, d) is said to converge, or to be convergent iff there is an 
x e X such that 

lim d(x n , x) — 0. (3.1) 

The element x is called the limit of (x n ) (i.e., limit of the sequence), and we may 
state that 

lim x n — x. (3.2) 
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We say that (x n ) converges to x or has a limit x. If (x n ) is not convergent, then 
we say that it is a divergent sequence, or is simply divergent. 

A shorthand expression for (3.2) is to write x n — > x. We observe that sequence (x„) 
is defined to converge (or not) with respect to a particular metric here denoted d 
(recall the axioms for a metric space from Chapter 1). We remark that it is possible 
that, for some (x„) in some set X, the sequence might converge with respect to 
one metric on the set, but might not converge with respect to another choice of 
metric. It must be emphasized that the limit x must be an element of X in order 
for the sequence to be convergent. 

Suppose, for example, that X — (0, 1] C R, and consider the sequence x n — 
■Ar{(n e Z + ). Suppose also that d(x, y) = \x — y\. The sequence (x n ) does not 
converge in X because the sequence "wants to go to 0." But is not in X. So the 
sequence does not converge. (Of course, the sequence converges in X — R with 
respect to our present choice of metric.) 

It can be difficult in practice to ascertain whether a particular sequence con- 
verges according to Definition 3.1. This is because the limit x may not be known 
in advance. In fact, this is almost always the case in computing applications of 
sequences. Sometimes it is therefore easier to work with the following: 

Definition 3.2: Cauchy Sequence, Complete Space A sequence (x n ) in a 
metric space X — (X, d) is called a Cauchy sequence iff for all 6 > there is an 
N{e) € Z+ such that 

d(x m ,x n ) < e (3.3) 

for all m, n > N(e). The space X is a complete space iff every Cauchy sequence 
in X converges. 

We often write N instead of N(e), because N may depend on our choice of e. It 
is possible to prove that any convergent sequence is also Cauchy. 

We remark that, if in fact the limit is known (or at least strongly suspected), 
then applying Definition 3.1 may actually be easier than applying Definition 3.2. 

We see that under Definition 3.2 the elements of a Cauchy sequence get closer to 
each other as n and m increase. Establishing the "Cauchiness" of a sequence does 
not require knowing the limit of the sequence. This, at least in principle, simplifies 
matters. However, a big problem with this definition is that there are metric spaces 
X in which not all Cauchy sequences converge. In other words, there are incomplete 
metric spaces. For example, the space X — (0, 1] with d(x, y) = \x — y\ is not 
complete. Recall that we considered x n = l/(« + 1). This sequence is Cauchy, 1 
but the limit is 0, which is not in X. Thus, this sequence is a nonconvergent Cauchy 
sequence. Thus, the space (X, | • |) is not complete. 



We see that 



d\Xm> x n) '■ 



1 1 



m + 1 n + 1 
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A more subtle example of an incomplete metric space is the following. Recall 
space C[a, b] from Example 1.4. Assume that a — and b = 1, and now choose 
the metric to be 



d(x, y) 



-I 

Jo 



\x(t) - y(t)\dt 



(3.4) 



instead of Eq. (1.8). Space C[0, 1] with the metric (3.4) is not complete. This 
may be shown by considering the sequence of continuous functions illustrated 
in Fig. 3.1. The functions x m {t) in Fig. 3.1a form a Cauchy sequence. (Here we 
assume m > 1, and is an integer.) This is because d(x m ,x n ) is the area of the 
triangle in Fig. 3.1b, and for any e > 0, we have 

V ttl ' Yl ) ^^ 

whenever m,n > l/(2e). (Suppose n >m and consider that d(x m , x n ) — j(— — 
) < j- < €.) We may see that this Cauchy sequence does not converge. Observe 
that we have 



x m (t) — for t € [0, j], x m (t) —I for t e [a m , 1], 





t 

Figure 3.1 A Cauchy sequence of functions in C[0, 1]. 

For any e > we may find 7V(e) > such that for n > m > N(e) 

1 1 
< e. 

m n 

If h < m, the roles of n and m may be reversed. The conditions of Definition 3.2 are met and so the 
sequence is Cauchy. 
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where a m — j + ■£-. Therefore, for all x e C[0, 1], 



d(x m ,x) — / \x m (t) -x(t)\dt 



/ \x„ 
Jo 

/■1/2 ra m /•< 

/ \x(t)\dt+ \x m (t)- x(t)\dt + 

JO J 1/2 Ja m 



•1/2 fa,„ n\ 

,(f) - x(t)\dt + I \\ - x(t)\dt. 



The integrands are all nonnegative, and so each of the integrals on the right-hand 
side are nonnegative, too. Thus, to say that d(x m , x) -> implies that each integral 
approaches zero. Since x(t) is continuous, it must be the case that 

x(t) = for t e [0, \), x(t) = 1 for t e (± 1]. 



However, this is not possible for a continuous function. In other words, we have 
a contradiction. Hence, (x n ) does not converge (i.e., has no limit in X — C[0, 1]). 
Again, we have a Cauchy sequence that does not converge, and so C[0, 1] with 
the metric (3.4) is not complete. 

This example also shows that a sequence of continuous functions may very well 
possess a discontinuous limit. Actually, we have seen this phenomenon before. 
Recall the example of the Fourier series in Chapter 1 (see Example 1.20). In this 
case the series representation of the square wave was made up of terms that are all 
continuous functions. Yet the series converges to a discontinuous limit. We shall 
return to this issue again later. 

So now, some metric spaces are not complete. This means that even though a 
sequence is Cauchy, there is no guarantee of convergence. We are therefore faced 
with the following questions: 

1. What metric spaces are complete? 

2. Can they be "completed" if they are not? 

The answer to the second question is "Yes." Given an incomplete metric space, 
it is always possible to complete it. We have seen that a Cauchy sequence does 
not converge when the sequence tends toward a limit that does not belong to the 
space; thus, in a sense, the space "has holes in it." Completion is the process of 
filling in the holes. This amounts to adding the appropriate elements to the set that 
made up the incomplete space. However, in general, this is a technically difficult 
process to implement in many cases, and so we will never do this. This is a job 
normally left to mathematicians. 

We will therefore content ourselves with answering the first question. This will 
be done simply by listing complete metric spaces that are useful to engineers: 

1. Sets R and C with the metric d(x, y) = \x — y\ are complete metric spaces. 
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2. Recall Example 1.3. The space /°°[0, oo] with the metric 

d(x,y) = sup \xk - yk\ 

keZ+ 



(3.5) 



is a complete metric space. (A proof of this claim appears in Ref. 1, p. 34.) 
3. The Euclidean space R" and the unitary space C" both with metric 



d(x,y) = 



-.1/2 



^2 \xk -ykY 



.k=0 



(3.6) 



are complete metric spaces. (Proof is on p. 33 of Ref. 1.) 
4. Recall Example 1.6. Fixing p, the space l p [0, oo] such that 1 < p < oo is a 
complete metric space. Here we recall that the metric is 



d(x, y) 



J2\xk -yk\' 



.k=Q 



UP 



(3.7) 



5. Recall Example 1.4. The set C[a, b] with the metric 
d(x,y)— sup \x(t) — y(t)\ 

t€[a,b] 

is a complete metric space. (Proof is on pp. 36-37 of Ref. 1.) 



(3.8) 



The last example is interesting because the special case C[0, 1] with metric (3.4) 
was previously shown to be incomplete. Keeping the same set but changing the 
metric from that in (3.4) to that in (3.8) changes the situation dramatically. 

In Chapter 1 we remarked on the importance of the metric space L 2 [a, b] (recall 
Example 1.7). The space is important as the "space of finite energy signals on the 
interval [a, b]." (A "finite power" interpretation was also possible.) An important 
special case of this was L 2 (R) = L 2 (— oo, oo). Are these metric spaces complete? 
Our notation implicitly assumes that the set (1.11a) (Chapter 1) contains the so- 
called Lebesgue integrable functions on [a,b]. In this case the space L 2 [a, b] is 
indeed complete with respect to the metric 



d(x, y) 



f 

.J a 



,1/2 



\x(t) - y(t)\ dt 



(3.9) 



Lebesgue integrable functions 2 have a complicated mathematical structure, and we 
have promised to avoid any measure theory in this book. It is enough for the reader 

One of the "simplest" introductions to these is Rudin [2]. However, these functions appear in the 
last chapter [2, Chapter 11]. Knowledge of much of the previous chapters is prerequisite to studying 
Chapter 11. Thus, the effort required to learn measure theory is substantial. 



TLFeBOOK 



68 SEQUENCES AND SERIES 

to assume that the functions in L 2 [a, b] are the familiar ones from elementary 
calculus. 3 

The complete metric spaces considered in the two previous paragraphs also 
happen to be normed spaces; recall Section 1.3.2. This is because the metrics are 
all induced by suitable norms on the spaces. It therefore follows that these spaces 
are complete normed spaces. Complete normed spaces are called Banach spaces. 

Some of the complete normed spaces are also inner product spaces. Again, this 
follows because in those cases an inner product is defined that induced the norm. 
Complete inner product spaces are called Hilbert spaces. To be more specific, the 
following spaces are Hilbert spaces: 

1 . The Euclidean space R" and the unitary space C" along with the inner product 

71-1 



y) = J2 x *y*k ( 3 - 10 ) 



k=0 



are both Hilbert spaces. 
2. The space L 2 [a, b] with the inner product 



( x ,y) = J x(t)y*(t)dt (3.11) 

J a 



is a Hilbert space. [This includes the special case L 2 (R).] 
3. The space Z 2 [0, oo] with the inner product 



k=0 



y) = J2 x ^y*k ( 3 - 12 ) 



is a Hilbert space. 



We emphasize that (3.10) induces the metric (3.6), (3.11) induces the metric (3.9), 
and (3.12) induces the metric (3.7) (but only for case p — 2; recall from Chapter 1 
that l p [0, oo] is not an inner product space when p ^ 2). The three Hilbert spaces 
listed above are particularly important because of the fact, in part, that elements in 
these spaces have (as we have already noted) either finite energy or finite power 
interpretations. Additionally, least-squares problems are best posed and solved 
within these spaces. This will be considered later. 

Define the set (of natural numbers) N = {1, 2, 3, . . .}. We have seen that sequences 
of continuous functions may have a discontinuous limit. An extreme example of this 
phenomenon is from p. 145 of Rudin [2]. 

These "familiar" functions are called Riemann integrable functions. These functions form a proper 
subset of the Lebesgue integrable functions. 
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Example 3.1 For n e N define 

x„(f) = lim [cos(n\jrt)] 2m . 

When n\t is an integer, then x n (t) — 1 (simply because cos(nk) = ±1 for k e Z). 
For all other values of t , we must have x n (t) — (simply because | cos t \ < 1 when 
t is not an integral multiple of jt). Define 

x(t) — lim x n (t). 

If t is irrational, then x„(f) = for all n. Suppose that t is rational; that is, suppose 
t — p/q for which p, q e Z. In this case «!f is an integer when n > q in which 
case x(t) — 1. Consequently, we may conclude that 

. . ,. ,• r / , s-,2m \ 0> t is irrational ,. ... 

x(t) — lim lim [cos(«!jrf)] = i , • • (3.13) 

n^Kuii^oo I 1, t is rational 

We have mentioned (in footnote 3, above) that Riemann integrable functions are a 
proper subset of the Lebesgue integrable functions. It turns out that x(t) in (3.13) 
is Lebesgue integrable, but not Riemann integrable. In other words, you cannot use 
elementary calculus to find the integral of x(t) in (3.13). Of course, x(t) is a very 
strange function. This is typical; that is, functions that are not Riemann integrable 
are usually rather strange, and so are not commonly encountered (by the engineer). 
It therefore follows that we do not need to worry much about the more general 
class of Lebesgue integrable functions. 

Limiting processes are potentially dangerous. This is illustrated by a very simple 
example. 

Example 3.2 Suppose n, m e N. Define 

m 

x m,n = ; • 



(This is a double sequence. In Chapter 1 we saw that these arise routinely in wavelet 
theory.) Treating n as a fixed constant, we obtain 



so 



lim x mn = 1 

m— >oo 



n^oo m^oo 



Now instead treat m as a fixed constant so that 

lim x m ,„ — 
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which in turn implies that 

Interchanging the order of the limits has given two completely different answers. 

Interchanging the order of limits clearly must be done with great care. 

The following example is simply another illustration of how to apply Defini- 
tion 3.2. 

Example 3.3 Define 



1 



(n e Z+). 



This is a sequence in the metric space (R, | • |). This space is complete, so we need 
not know the limit of the sequence to determine whether it converges (although 
we might guess that the limit is x — 1). We see that 



iX \ytfti j •™yi ) 



where the triangle inequality has been used. If we assume [without loss of generality 
(commonly abbreviated w.l.o.g.)] that n > m > N(e) then 



(-l) m 


(-1)" 


1 111 






< 1 < \- - 


m + 1 


n + 1 


m + 1 n + 1 m n 



1 




1 




2 






— 


+ 


— 


< 


— 


< 


€. 


in 




n 




in 







So, for a given e > 0, we select n > m > 2/e. The sequence is Cauchy, and so it 
must converge. 

We close this section with mention of Appendix 3. A. Think of the material in it 
as being a very big applications example. This appendix presents an introduction 
to coordinate rotation digital computing (CORDIC). This is an application of a 
particular class of Cauchy sequence (called a discrete basis) to the problem of 
performing certain elementary operations (e.g., vector rotation, computing sines 
and cosines). The method is used in application-specific integrated circuits (ASICs), 
gate arrays, and has been used in pocket calculators. Note that Appendix 3. A also 
illustrates a useful series expansion which is expressed in terms of the discrete basis. 

3.3 POINTWISE CONVERGENCE AND UNIFORM CONVERGENCE 

The previous section informed us that sequences can converge in different ways, 
assuming that they converge in any sense at all. We explore this issue further here. 
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Definition 3.3: Pointwise Convergence Suppose that (x n (t)) (n e Z + ) is a 
sequence of functions for which t e S C R. We say that the sequence converges 
pointwise iff there is an x(t) (t e S) so that for all e > there is an N — N(e, t) 
such that 

\x n (t) - x(t)\ < € (3.14) 

for n > N. We call x the limit of (x n ) and write 

x(t) = lim x„(f) (f € S). (3.15) 

n— »-00 

We emphasize that under this definition A' may depend on both e and t. We may 
contrast Definition 3.3 with the following definition. 

Definition 3.4: Uniform Convergence Suppose that (x n (t)) (n e Z + ) is a 
sequence of functions for which t e S C R. We say that the sequence converges 
uniformly iff there is an x(t) (t e S) so that for all e > there is an Af = A^(e) 
such that 

\x n (t)-x(t)\<e (3.16) 

for n > N. We call x the limit of (x n ) and write 

x(t) = lim x„{t) (t e S). (3.17) 

n— >oo 

We emphasize that under this definition N never depends on t, although it may 
depend on e. It is apparent that a uniformly convergent sequence is also point- 
wise convergent. However, the converse is not true; that is, a pointwise convergent 
sequence is not necessarily uniformly convergent. This distinction is important in 
understanding the convergence behavior of series as well as of sequences. In par- 
ticular, it helps in understanding convergence phenomena in Fourier (and wavelet) 
series expansions. 

In contrast with the definitions of Section 3.2, under Definitions 3.3 and 3.4 the 
elements of (x n ) and the limit x need not reside in the same function space. In 
fact, we do not ask what function spaces they belong to at all. In other words, the 
definitions of this section represent a different approach to convergence analysis. 

As with the Definition 3.1, direct application of Definitions 3.3 and 3.4 can be 
quite difficult since the limit, assuming it exists, is not often known in advance 
(i.e., a priori) in practice. Therefore, we would hope for a convergence criterion 
similar to the idea of Cauchy convergence in Section 3.2 (Definition 3.2). In fact, 
we have the following theorem (from Rudin [2, pp. 147-148]). 

Theorem 3.1: The sequence of functions (x n ) defined on S C R converges 
uniformly on S iff for all e > there is an N such that 

\x m {t) - x n {t)\ < e (3.18) 

for all n,m > N. 
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This is certainly analogous to the Cauchy criterion seen earlier. (We omit the 
proof.) 



Example 3.4 Suppose that (x„) is defined according to 



x n (t) = 



1 



nt + 1 



t € (0, 1) and n e N. 



A sketch of x„(t) for various n appears in Fig. 3.2. We see that ("by inspection") 
x n -> 0. But consider for all e > 



\x„(t)-0\ 



1 



nt + 1 



< e 



which implies that we must have 



B >I(I-l)=tf 

so that N is a function of both t and e. Convergence is therefore pointwise, and is 
not uniform. 

Other criteria for uniform convergence may be established. For example, there is 
the following theorem (again from Rudin [2, p. 148]). 



Theorem 3.2: Suppose that 



Define 



lim x n (t) = x(t) (t e S). 



M n = sup\x„(t) - x(t)\. 
teS 



Then x n — > x uniformly on S iff M„ — > as n — > oo. 
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Figure 3.2 A plot of typical sequence elements for Example 3.4; here, t e [0.01, 0.99]. 
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8 10 



Figure 3.3 A plot of typical sequence elements for Example 3.5. 



The proof is really an immediate consequence of Definition 3.4, and so is omitted 
here. 



Example 3.5 Suppose that 



x„(t) 



t 



t e R and n e N. 



1 + nt 2 ' 
A sketch of x n (t) for various n appears in Fig. 3.3. We note that 

dx n (t) (1 + nt 2 ) ■ 1 - t ■ (2nt) 1 - nt 2 



dt 



[l+nt 1 ] 



2l2 



[1+Hf 2 ] 



2l2 



for t = ±-7=. We see that 

sin 



4n) Isfn 

We also see that x n — >• 0. So then 

1 
M n = sup|x„0)| = t— =■ 

Clearly, M„ — >■ as n — >■ oo. Therefore, via Theorem 3.2, we immediately conclude 
that x n — > x uniformly on the real number line. 



3.4 FOURIER SERIES 

The Fourier series expansion was introduced briefly in Chapter 1, where the behav- 
ior of this series with respect to its convergence properties was not mentioned. In 
this section we shall demonstrate the pointwise convergence of the Fourier series 
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by the analysis of a particular example. Much of what follows is from Walter [17]. 
However, of necessity, the present treatment is not so rigorous. 
Suppose that 

(3.19) 



,'(f) = — (7T - t), < t < 2lt. 
Tt 



The reader is strongly invited to show that this has Fourier series expansion 

2 °° 1 
git) = - J] - sm(kt). (3.20) 



% ' — ' k 
k=\ 



The procedure for doing this closely follows Example 1.20. In our analysis to 
follow, it will be easier to work with 



fit) = -zgit) =y 7 sin(^f). 

2 * — ' k 

k=i 

Define the sequence of partial sums 

n 1 

S n (t) — V* - &in(kt) (n e N). 

' ' h 

So we infer that 



(3.21) 



k=\ 



lim S n it) = fit), 



(3.22) 



(3.23) 



but we do not know in what sense the partial sums tend to f{t). Is convergence 
pointwise, or uniform? 

We shall need the special function 



D n it)=- 

Tt 



- + Y^ cos i kt ) 



k=\ 



1 sin(« + 7j)f 
2tt sin(if) 



(3.24) 



This function is called the Dirichlet kernel. The second equality in (3.24) is not 
obvious. We will prove it. Consider that 

-t\ inD n it))= 2 Sin l2/ +IZ sin (2 f ) COs(A:f) 

v 7 k=\ v 



t + - y] sin | - - A- 1 / 



sin ( fe If, 

(3.25) 
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where we have used the identity sin a cosb — j sin(a + b) + j sin(a — b). By 
expanding the sums and looking for cancellations 



k=\ v 7 k=\ v ' 

Applying (3.26) in (3.25), we obtain 



X \ ( X 

sin | n H — J t — sin I —t 



(3.26) 



sin ( -t 1 (jrD n (t)) = - sin I n + - 1 / 



so immediately 



D n it) 



1 sin(« + j)f 
2tt sin(if) 



and this establishes (3.24). Using the identity sin(a + b) = sin a cosZ? + cos a sinfe, 
we may also write 



D n (t) 



1 

2tt 



sin(nf)cos(jf) 
sin(lf) 



+ cos(«f) 



(3.27) 



For t > 0, using the form of the Dirichlet kernel in (3.27), we have 

n J D n (x)dx — I 
Jo Jo 



sin(nx) cos(ix) 1 

: — 1 — cos(nx) 

2sin(4x) 2 



dx 



= f'sm(nx ldx+ r' 
Jo x J 



1 cos(^x) 1 

2 sin(±x) x 



dx 



cos(nx) dx. 



(3.28) 



We are interested in what happens when t is a small positive value, but n is large. 
To begin with, it is not difficult to see that 



Less clearly 



lim 

n— >oo 



2 7o 

Jo 



1 1 

cos(«x) dx — lim — sin(nf) = 0. 

n^oo 2n 



lim / sin(nx) 



1 cos(^x) 1 

2 sin(±x) x 



dx — 



(3.29) 



(3.30) 
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(take this for granted). Through a simple change of variable 

sin(«:t) 



f' sin(«:t) f 

l(nt) = / — - — -dx = / 
Jo x Jo 



dx. 



In fact 



f°° sinx tc 

I dx — — . 

Jo x 2 



(3.31) 



(3.32) 



This is not obvious, either. The result may be found in integral tables [18, p. 483]. 
In other words, even for very small t, I(nt) does not go to zero as n increases. 
Consequently, using (3.29), (3.30), and (3.31) in (3.28), we have (for big n) 



n I D n (x 
Jo 



) dx ss lint). 



(3.33) 



The results in the previous paragraph help in the following manner. Begin by 
noting that 



" 1 n rt ft 

S n (t) — 2_\ — sia(kt) — 2_, I cos(£x) dx — I 



y^ cos(kx) 



.k=\ 



dx 



f 

Jo 



1 v^ 

- + 2_^cos(£x) 



k=\ 



dx 



1 f 1 

-t = n D n {x)dx--t (via (3.24)). 

2 Jo 2 



So from (3.33) 



5„(0« I(nt)- if. 



Define the sequence t n — -jr. Consequently 



1 



S n (t„) « I in)- —IT. 
In 

As n — > oo, t n -> 0, and S„(t n ) — > /(jt). We can say that for big n 

S„(0+) « 7(7T). 

Now, /(0+) = |-, so for big n 
5»(0+) 



/(0+) 



2 r 

* Jo 



■dx % 1.18. 



(3.34) 
(3.35) 

(3.36) 
(3.37) 

(3.38) 



Numerical integration is needed to establish this. This topic is the subject of a later 
chapter, however. 

We see that the sequence of partial sums S„ (0+) converges to a value bigger 
than /(0+) asn^ oo. The approximation S n (t) therefore tends to "overshoot" 
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the true value of fit) for small t . This is called the Gibbs phenomenon, or Gibbs 
overshoot. We observe that t — is the place where f(t) has a discontinuity. This 
tendency of the Fourier series to overshoot near discontinuities is entirely typical. 
We note that f(n) — S n (jt) — for all n > 1. Thus, for any e > 

\f(it)-S n (jt)\<e 

for all n > 1. The previous analysis for t — 0+, and this one for t — n show that 
N (in the definitions of convergence) depends on t. Convergence of the Fourier 
series is therefore pointwise and not uniform. Generally, the Gibbs phenomenon is 
a symptom of pointwise convergence. 

We remark that the Gibbs phenomenon has an impact in the signal processing 
applications of series expansions. Techniques for signal compression and signal 
enhancement are often based on series expansions. The Gibbs phenomenon can 
degrade the quality of decompressed or reconstructed signals. The phenomenon 
is responsible for "ringing artifacts." This is one reason why the convergence 
properties of series expansions are important to engineers. 

Figure 3.4 shows a plot of /(f), S n (t) and the error 



E n (t) = S n (t)-f(t). 



(3.39) 



The reader may confirm (3.38) directly from the plot in Fig. 3.4a. 
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Figure 3.4 Plots of the Fourier series expansion for f(t) in (3.21), S n (t) [of (3.22)] for 
n = 30, and the error E n (t) = S n (t) — fit). 
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Figure 3.5 E n (t) of (3.39) for different values of n. 



We conclude this section by remarking that 



lim 



/>2tt 
JO 



\E n (t)\ 2 dt = 0, 



(3.40) 



that is, the energy of the error goes to zero in the limit as n goes to infinity. 
However, the amplitude of the error in the vicinity of a discontinuity remains 
unchanged in the limit. This is more clearly seen in Fig. 3.5, where the error is 
displayed for different values of n. Of course, this fact agrees with our analysis. 
Equation (3.40) is really a consequence of the fact that (recalling Chapter 1) S n (t) 
and fit) are both in the space L 2 (0, 2it). Rigorous proof of this is quite tough, 
and so we omit the proof entirely. 



3.5 TAYLOR SERIES 

Assume that fix) is real-valued and that x e R. One way to define the derivative 
of f{x) at x — xq is according to 



f w ix) 



df(x) , 



— — lx=X0 = lim ■ 
ax x^*o 



f(x) - fix Q ) 



Xq 



The notation is 



f in) ix) 



d"fix) 



dx n 



(so f (U) ix) = f(x)). From (3.41), we obtain 



./■(x)«/(xo) + / (1) (xo)(x-xo). 



(3.41) 



(3.42) 



(3.43) 



But how good is this approximation? Can we obtain a more accurate approximation 
to fix) if we know f^ n \xo) for n > 1? Again, what is the accuracy of the resulting 
approximation? We consider these issues in this section. 
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Begin by recalling the following theorem. 

Theorem 3.3: Mean- Value Theorem If f(x) is continuous for x e [a, b] 

with a continuous derivative for x e (a, b), then there is a number f € (a, b) such 

that 

fib) - /(a) m 

: =/ (1) (£). (3.44) 

o — a 

Therefore, if a = xo and b — x, we must have 

/(x) = /(x ) + / (1) (f)(x-x ). (3.45) 

This expression is "exact," and so is in contrast with (3.43). Proof of Theo- 
rem 3.3 may be found in, for example, Bers [19, p. 636]. Theorem 3.3 generalizes 
to the following theorem. 

Theorem 3.4: Generalized Mean- Value Theorem Suppose that f(x) and 
g(x) are continuous functions on x e [a, b]. Assume /^(x), and g^ix) exist and 
are continuous, and g^\x) ^ for x e (a, b). There is a number f € (a, b) such 
that 

f{b) - /(a) / (1) (f) 



m-gia) £«(£) 



(3.46) 



Once again the proof is omitted, but may be found in Bers [19, p. 637]. 

The tangent to f(x) at x — xo is given by t(x) — f(xo) + / (1) (^o)U — xq) 
(f (xo) — /(xo) and f^(xo) = f^Hxo))- We wish to consider t(x) to be an approx- 
imation to f(x), so the error is 

f(x) -t(x) = e(x) 

or 

f(x) = /(x ) + f m (x )(x - x ) + e(x). (3.47) 

/ (1) (^o). 



Thus 



e(x) f{x) - /(x ) ^. (1) 





x — xo x — xo 


But 


fix) - /(xo) 
hm = / 




*->*o x — xq 


so immediately 


eix) 
lim — — - = 0. 



Z (1) (*o) 



(3.48) 

*^*0 X — Xo 

From (3.47) e(xo) = 0, and also from (3.47), we obtain 

f m (x) = f m ix Q ) + e m ix), (3.49) 
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so e ( - r> (xo) — 0. From (3.49), f^Hx) — e^ 2 '(x) (so we now assume that f(x) has 
a second derivative, and we will also assume that it is continuous). Theorem 3.4 
has the following corollary. 

Corollary 3.1 Suppose that f(a) — g(a) — 0, so for all b ^ a there is a £ € 
(a, b) such that 

fib) f m (H) 



8(b) g m 0;) 



(3.50) 



Now apply this corollary to f(x) — e(x), and g(x) — (x — xo) 2 , with a — xq, 
b — x. Thus, from (3.50) 

e(x) e m (t) 

(3.51) 



(x - xo) 1 2(t - x ) 

(t € (xq, x)). Apply the corollary once more to f(x) = e^'(x), and g(r) 
2(t - x ): 

e (V) (r) 1 m 1 n\ 



2(r - x ) 2 
(£ € (*<>, T)). Apply (3.51) in (3.52) 



= ^'(?) = ^/ w g) (3-52) 



«W ! .(2), 



= t/ w «) 



(x-x ) 2 2' 
or (for some £ € (xo, x)) 

e(*) = |/ (2) (?)(x-x ) 2 . (3.53) 

If \f (2) (t)\ < M 2 for t e (x , x), then 

\e(x)\ <^-j-M 2 |x-x | 2 (3.54) 

and 

/(x) = /(xo) + / (1) (x )(x - xq) + e(x), (3.55) 

for which (3.54) is an upper bound on the size of the error involved in approxi- 
mating f(x) using (3.43). 

Example 3.6 Suppose that f(x) — *J~x; then 



Suppose that xo = 1, and x = xo + Sx — 1 + Sx. Thus, via (3.55) 

Vx" = Vl+5x = 1 + / (1) (l)5x + e(x) = 1 + ±5x + e(x), 



TLFeBOOK 



TAYLOR SERIES 81 

so if, for example, \Sx\ < |, then \f (2) (x)\ < 2 (for x e (\, |)) so M 2 = 2, 
and so 

\e(x)\ < (Sx) 2 

via (3.54). This bound may be compared to the following table of values: 







1 + \Sx 


e(x) 




Sx 


VI + <5x 


(Sx) 2 


3 

4 


0.5000 


0.6250 


-0.1250 


0.5625 


1 

2 


0.7071 


0.7500 


-0.0429 


0.2500 





1.0000 


1.0000 


0.0000 


0.0000 


1 

2 


1.2247 


1.2500 


-0.0253 


0.2500 


3 
4 


1.3229 


1.3750 


-0.0521 


0.5625 



It is easy to see that indeed \e(x)\ < (Sx) 2 . 

We mention that Corollary 3.1 leads to l'Hopital's rule. It therefore allows us 
to determine 

.. /(*) 
hm 

x^a g( x ) 

when f(a) — g(a) — 0. We now digress briefly to consider this subject. The rule 
applies if f(x) and g(x) are continuous at x — a, if f(x) and g(x) have continu- 
ous derivatives at x = a, and if g^Hx) ^ near x — a, except perhaps at x = a. 

f m (x) 

l'Hopital's rule is as follows. If lim x ^ fl f(x) = lim^^ a g(x) — and lim x ^ a J {lj ^ 
exists, then 

f(x) f m (x) 

hm —— = hm . (3.56) 

x-*a g(x) x^a g( l >(x) 

The rationale is that from Corollary 3.1 for all x ^ a there is a £ € (a,b) such that 

fix) f^'(£) f*^(£) 

j±£ = )(1) .: . . So, if x is close to a then £ must also be close to a, and J (1) ; 
is close to its limit. l'Hopital's rule is also referred to as "the rule for evaluating 
the indeterminate form §." If it happens that f^\a) — g^(a) — 0, then one may 

attempt l'Hopital's rale yet again; that is, if lim x ^ a J (2) exists, then 



<■" g(2) (x) 



,. fix) .. f (2) (x) 
hm = hm —pr- . 

x^a g(x) x^a g( l >(x) 

Example 3.7 Consider 



im 

^0 


sinx 


-e x 

X 2 


+ 1 


= lim 

x^0 


d 

dx 


[sinx — 

d 
dx 


e x 

• 2 1 


+ 1] 










— lim 

x^0 


cosx — e x 
2x 


= 


lim - 

x^0 



sinx — e x 1 

2 = ~2 
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for which the rule has been applied twice. Now consider instead 



1 — 2x j r 

lim = lim " ' 



— [1 -2x] 



2 + 4* ^od_ [2 + Ax] 
dx 



lim — 

x^o 4 



1 



This is wrong ! l'Hopital's rule does not apply here because /(0) = 1, and g(0) = 2 
(i.e., we do not have /(0) = g(0) — as needed by the theory). 

The rule can be extended to cover other indeterminate forms (e.g., — ). For example, 
consider 



1 

log x Z 

lim x log. x — lim — - — = lim 
x^-0 x^O 1 x^o 1 



= lim (-X) = 0. 

x^Q 



An interesting case is that of finding 



lim 1 + - 

x^oa \ x 



This is an indeterminate of the form 1°°. Consider 



lim log I 1 + - 

x^oo \ x 



lim 

x— >oo 



log, I 1 + - 

I 

x 
1 / 1 



1 



lim 



lim 



1 + - 

X 



The logarithm and exponential functions are continuous functions, so it happens to 
be the case that 



I lim log e I 1 + - ) -= log e 



lim I 1 + - 

*^°o \ x 
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that is, the limit and the logarithm can be interchanged. Thus 

= ■^LssoH)*] 



e l = 



so finally we have 



H)' 



lim 1 + - = e. (3.57) 



More generally, it can be shown that 



lim 

n— »oo 



(X\ n 
1 + -) = e*. (3.58) 



This result has various applications, including some in probability theory relating to 
Poisson and exponential random variables [20]. An alternative derivation of (3.57) 
appears on pp. 64-65 of Rudin [2], but involves the use of the Maclaurin series 
expansion for e. We revisit the Maclaurin series for e x later. 
We have demonstrated that for suitable £ € (xq, x) 

fix) = f(x ) + f m (x )(x - x ) + \f {2 \H )(x - xo) 2 

(recall (3.55)). Define 

p(x) = /(xo) + f m (x )(x - xo) + y (2) (x )(x - xo) 2 (3.59) 

so this is some approximation to f(x) near x — xo- Equation (3.43) is a linear 
approximation to f(x), and (3.59) is a quadratic approximation to f(x). Once 
again, we wish to consider the error 

f(x) - p(x) = e(x). (3.60) 

We note that 

p(x ) = f(x ), p m (x Q ) = f m (xo), p (2 \xo) = f {2 \xo). (3.61) 

In other words, the approximation to f(x) in (3.59) matches the function and its 
first two derivatives at x — xq. Because of (3.61), via (3.60) 

e(x ) = e m (xo) = e (2) (x ) = 0, (3.62) 

and so via (3.59) and (3.60) 

e (3) (x) = / (3) (x) (3.63) 
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(because // 3 '(x) — since p(x) is a quadratic in x). As in the derivation of (3.53), 
we may repeatedly apply Corollary 3.1: 



e(x) e^{h) 



(x-x ) 3 3(ti-x ) 2 
eW(ti) e (2) (?2) 



3(fi-x ) 2 3-2(t 2 -x ) 
e (2) (t 2 ) e (3) (f) 



3 • 2(t 2 -x Q ) 3-2 
which together yield 

e{x) _ /< 3 >(g) 
(x — xo) 3 3 • 2 

or 



for t\ e (xo, x) 
for ? 2 e (x , t\) 
for f € (x ,r 2 ), 



for £ € (xo, x) 



«(*) = ^-y / (3) (f X* - xo) 3 (3.64) 

for some f € (xo, x). Thus 

/(*) = /(xo) + / (1) (xo)(x - xo) + — / (2) (x )(x - xo) 2 + e(x). (3.65) 

Analogously to (3.54), if \f®\t)\ 5- ^3 f° r t e (xo,x), then we have the error 
bound 

k(x)|<^-j-M 3 |x-xo| 3 . (3.66) 

We have gone from a linear approximation to /(x) to a quadratic approximation 
to f(x). All of this suggests that we may generalize to a degree n polynomial 
approximation to /(x). Therefore, we define 

n 

Pn(x) = ^2p„,k(x - x ) k , (3.67) 

k=Q 



where 

r>„ 7, = 

k\ 

Then 



Pn,k = A/ (t) (*o). (3-68) 



/(x) = p„(x) + e„+i(x), (3.69) 

where the error term is 



e n+1 (x) = -_L_/(»+l>($)(jt -x )" +1 
(n + 1)! 



(3.70) 
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for suitable £ € (xo,x). We call p n {x) the Taylor polynomial of degree n. This 
polynomial is the approximation to /(x), and the error e n +\(x) in (3.70) can be 
formally obtained by the repeated application of Corollary 3.1. These details are 
omitted. Expanding (3.69), we obtain 

fix) = f(xo) + f m (x )(x - xo) + ^/ (2) (x )(x - x ) 2 

+ ■■■ + -J (n \x*){x - x ) n + — { ^-J in+l \H){x - x )" +1 (3.71) 
n ! (n + 1 ) ! 

which is the familiar Taylor formula for f(x). We remark that 

f ik \xo) = p?\xo) (3.72) 

for k = 0, 1, . . . , n — 1, n. So we emphasize that the approximation p n (x) to f(x) 
is based on forcing p„(x) to match the first n derivatives of f(x), as well as 
enforcing p n (xo) — f(xo). If \f^ n+v> {t)\ < M n+ \ for all t e I [interval / contains 
(xo, x)], then 

\e n +i(x)\ < 7— "-ttt M n+l \x - x \ n+1 . (3.73) 

If all derivatives of /(x) exist and are continuous, then we have the Taylor series 
expansion of f(x), namely, the infinite series 

/w = E 7^ /( " )(xo)(x - xo)A: - (3 - 74) 

fc=0 
The Maclaurin series expansion is a special case of (3.74) for xq — 0: 

/(*) = Yi -f (k \0)x k . (3.75) 

If we retain only terms k — to k — n in the infinite series (3.74) and (3.75), we 
know that e n+ \{x) gives the error in the resulting approximation. This error may 
be called the truncation error (since it arises from truncation of the infinite series 
to a finite number of terms). Now we consider some examples. 
First recall the binomial theorem 

(a+x) n = ^r( n k \x k a"- k , (3.76) 

where 



n 



k J k\{n-k)\' 



(3.77) 
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In (3.76) we emphasize that n e Z + . But we can use Taylor's formula to obtain an 
expression for (a + x) a when a^O, and a is not necessarily an element of Z + . 
Let us consider the special case 

f(x) = (l+x) a 

for which (if k > 1) 

f (k) (x) = a(a - l)(a - 2) ■ ■ ■ (a - k + 1)(1 + x) a ~ k . (3.78) 

These derivatives are guaranteed to exist, provided x > — 1. We will assume this 
restriction always applies. So, in particular 

/ (fc) (0) = a(a - l)(a - 2) • • • (a - fc + 1) (3.79) 

giving the Maclaurin expansion 



" 1 
(1 + x f = 1 + ^ ry[«(« -!)■■■(«-*+ 1)]j 






[o(a- !)• ••(«-«)](! +f) a -"- 1 x n+1 (3.80) 



(n+1)! 
for some £ € (xo, x). We may extend the definition (3.77), that is, define 

o) = 1, (*) = ^ a(a ~ 1) --- (a ~ k + l)(k - l) (3 ' 81) 

so that (3.80) becomes 

tt +*)" = £(" )**+(„ + ! )a + §) a "" _1 *" +1 , (3.82) 

k=0 , 



--Pn(x) =««+!« 



for x > — 1 . 



Example 3.8 We wish to compute [1.03] 1 / 3 with n — 2 in (3.82), and to esti- 
mate the error involved in doing so. We have x — 0.03, a — i, and £ e (0, .03). 
Therefore from (3.82) [1 + X] 1 / 3 is approximated by the Taylor polynomial 

p 2 ( X ) = 1 + ( J J X + f 3 \ x 2 = l + ^ x _^ x 2 
SO 

[1.03] 1/3 « £> 2 (0.03) = 1.009900000 
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but [1.03] 1/3 = 1.009901634, so e 3 (x) = 1.634 x 10" 6 . From (3.82) 

and so e 3 (0.03) = Jr^r(l + £)~ 8/3 = | x 10" 6 (1 + £)" 8/3 . Since < £ < 0.03, 
we have 

1.5403 x 10" 6 < e 3 (.03) < 1.6667 x 10" 6 . 

The actual error is certainly within this range. 

If f(x) = -A-r, and if xo — 0, then 



l+x 

n+l„n+l 
1 +X 



1 " 



***+ ( } X . (3.83) 



W=-^ («^1). (3.84) 

■^^ 1 — a 



=r(*) 

This may be seen by recalling that 

l-a" +1 

y* = 

ifc=0 

so ELo(-i)^ = 1 ~ ( ~ 1 1 ) +t'"" +1 - and thus 

A ^ ^ (-l) n+1 jc B+1 _ 1 - (-l) n+1 x n+1 (-l) n+1 x n+1 __ 1 
{-" X l+x l + x l+x l+x' 

k—0 

This confirms (3.83). We observe that the remainder term r(x) in (3.83) is not given 
by e„ + i (x) in (3.82). We have obtained an exact expression for the remainder using 
elementary methods. 

Now, from (3.83), we have 

1 -i_, + ,2_,3 + ... + ( _ ir -y-i , (-1)"'" 



l+t 1+t 

and we see immediately that 



f x dt , , , 

log e (l +x)= I = x - -x 1 + -x 3 

Jo l + t 



1 , 1 

— ; 
2 



-\ v n rx *n 



<—Y) n ~X n 

- — + (-1)"/ - — dt. (3.85) 



Jo l + t 



=r(x) 
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For x > 0, and < t < x we have -A^ < 1, implying that 



px jti px r"+l 

< / dt < t"dt = (x > 0). 

Jo 1 + 1 Jo n+l 



For —1 < x < with x < t < 0, we have 



1 1 

< 



l+t l+x 1-1*1 



so 



Jr 



/ -— dt <^^- / f" 

Jo 1 + f ~ 1-|*| Jo 
Consequently, we may conclude that 

1 

\r(x)\ < 



1 



1-1*1 



„ra+l 



« + 1 



IB+1 



(l-|x|)(« + l) 



n+l 



n + 1 



1*1 



n + l 



I (1 - |jc|)(n + 1) 



x > 

-1 <x <0 



(3.86) 



Equation (3.85) gives us a means to compute logarithms, and (3.86) gives us a 
bound on the error. 

Now consider (3.83) with x replaced by x 2 : 



1 + X k=Q 



{-\) n+i x 



n+\ v 2n+2 



l+X 2 



(3.87) 



Replacing n with n — 1, replacing x with f, and expanding, this becomes 
1 



1 - t 2 + t 4 



l+t 2 
where, on integrating, we obtain 



+ (_l)»-l,2»-2+ ( l)nt2n 



l + t 2 



tan x 



f x dt 1 
— I r- — x x 

Jo l + t 2 3 



3,5 7 

" ' -X X 



(_iyi-\ x 2n-\ 



2/i — l jq 



rx j.2n 

Jo TT7 



-dt . 



(3.S 



=r(*) 



Because -r-^j < 1 for all t e R, it follows that 



\r(x)\ < 



|2n+l 



2« + 1 



(3.89) 
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We now have a method of computing n. Since j — tan _1 (l), we have 

Tt 111 (-1)"" 1 



and 



r(1) -^T- (3 ' 91) 

2m + 1 



Using (3.90) to compute it is not efficient with respect to the number of arith- 
metic operations needed (i.e., it is not computationally efficient). This is because 
to achieve an accuracy of about 1/n requires about n/2 terms in the series [which 
follows from (3.91)]. However, if x is small (i.e., close to zero), then series (3.88) 
converges relatively quickly. Observe that 

tan" 1 — — — =tan _1 x + tan _1 y. (3.92) 

1 — xy 

Suppose that x = j, and y = A, then x ^ t — 1, so 

% = tan" 1 ( - ) + tan" 1 ( - ) . (3.93) 



4 V2. 

It is actually faster to compute tan _1 (j), and tan _1 (|) using (3.88), and for these 
obtain jt using (3.93) than to compute tan _1 (l) directly. In fact, this approach (a 
type of "divide and conquer" method) can be taken further by noting that 

tan" 1 ( - | = tan" 1 | - 1 + tan" 1 ( - | , tan" 1 ( - | = tan" 1 ( - | + tan" 1 



implying that 



— = 2 tan" 1 ( - | +tan _1 ( - 
4 \5 \7 



= 2tan _1 I - I +tan _1 I - + 2 tan -1 I - I . (3.94) 



Now consider f(x) — e x . Since f^(x) — e x for all k € Z + we have for xq — 
the Maclaurin series expansion 



oo k 

e*^-. (3.95) 

k=0 

This is theoretically valid for — oo < x < oo. We have employed this series before 
in various ways. We now consider it as a computational tool for calculating e x . 
Appendix 3.C is based on a famous example in Forsythe et al. [21, pp. 14-16]. 
This example shows that series expansions must be implemented on computers 
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with rather great care. Specifically, Appendix 3.C shows what can happen when we 
compute e~ 20 by the direct implementation of the series (3.95). Using MATLAB 
as stated e~ 20 «s 4.1736 x 10~ 9 , which is based on keeping terms k — to 88 
(inclusive) of (3.95). Using additional terms will have no effect on the final answer 
as they are too small. However, the correct value is actually e~ 20 — 2.0612 x 10 -9 , 
as may be verified using the MATLAB exponential function, or using a typical 
pocket calculator. Our series approximation has resulted in an answer possessing 
no significant digits at all. What went wrong? Many of the terms in the series 
are orders of magnitude bigger than the final result and typically possess rounding 
errors about as big as the final answer. The phenomenon is called catastrophic 
cancellation (or catastrophic convergence). As Forsythe et al. [21] stated, "It is 
important to realize that this great cancellation is not the cause of error in the 
answer; it merely magnifies the error already present in the terms." Catastrophic 
cancellation can in principle be eliminated by carrying more significant digits in the 
computation. However, this is costly with respect to computing resources. In the 
present problem a cheap and very simple solution is to compute e 20 using (3.95), 
and then take the reciprocal, i.e., use e~ 2Q — 1/e 20 . 
An important special function is the gamma function: 



-f 

Jo 



r(z)= / x z ~ l e~ x dx. (3.96) 

>0 

Here, we assume z € R. This is an improper integral so we are left to wonder if 

-M 



/ 

1 JO 



lim / x z e x dx 



exists. It turns out that the integral (3.96) converges for z > 0, but diverges for 
z < 0. The proof is slightly tedious, and so we will omit it [22, pp. 273-274]. If 
z — n e N, then consider 



/•OO 

{n) = / x" 
Jo 



r(n)= / x n ~ l e~ x dx. (3.97) 



Now 



r(n+l)=/ x n e~ x dx= lim / x n e~ x dx 

JO M^ooJq 

= lim -x n e- x \^+n x n - l e~ x dx 

M^oo |_ Jo 

(via f u dv — uv — j v du, i.e., integration by parts). Therefore 

/•OO 

r(n + l) = re/ x n - l e~ x dx =nT{n). (3.98) 

Jo 
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We see that T(l) = f™ e~ x dx = [-e~ x ]™ = 1. Thus, T(n + 1) = re!. Using the 
gamma function in combination with (3.95), we may obtain Stirling' s formula 

re! « ^/2nn n+1/2 e- n (3.99) 

which is a good approximation to re ! if re is big. The details of a rigorous derivation 
of this are tedious, so we give only an outline presentation. Begin by noting that 

/•OO rOO 

re!= / x n e~ x dx = \ e nXax - x dx. 
Jo Jo 

Let x — n + y, so 

/OO 
e nXain+y) - y dy. 
-n 

Now, since re ln(re + j) = n In [re (l + ^)] = re lnre + re In (l + ^), we have 

/oo 
e " ln "+" ]n (i+5)-yrfy 
-n 

/oo 
e „ ln( i + Z)_ y ^ 
-« 

Using (3.85), that is 

(i + Z) = Z_2l + Z_..., 

V re/ re 2re 2 3re 3 



In 
we have 



rein (l + -)->/ = - — 



2 3 

r , r 



2re 3n 2 
so 



re!=re"e-" / 

If now y = *Jnv, then dy = *Jndv, and so 

/oo 

So if re is big, then 



OO y 2 ^3 



OO 1,2 w 3 

re!=re" + Je"" / e~~ + ^~'" dv. 



■I. 



OO 2 



re! % re" + 2e " / e ? dv 
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If we accept that 



/ 

j — ( 



then immediately we have 



~ x2/2 dx = Jhi, 



/^ n-\- ~ —n 

'2jtn 2g " 



(3.100) 



and the formula is now established. Stirling's formula is very useful in statistical 
mechanics (e.g., deriving the Fermi-Dirac distribution of fermion particle energies, 
and this in turn is important in understanding the operation of solid-state electronic 
devices at a physical level). 

Another important special function is 



,'(*) 



1 



V2W 



exp 



(x — m) 
2^ 



2-\ 



-oo < X < oo 



(3.101) 



which is the Gaussian function (or Gaussian pulse). This function is of immense 
importance in probability theory [20], and is also involved in the uncertainty princi- 
ple in signal processing and quantum mechanics [23]. A sketch of g(x) for m — 0, 
with a 2 — 1, and a 2 = 0.1 appears in Fig. 3.6. For m — and a 2 — 1, the standard 
form pulse 

f(x) = -^=e- x2 ' 2 (3.102) 



'lit 



is sometimes defined [20]. In this case we observe that g(x) — — f [ x — ! ^-). We will 
show that 

(3.103) 



/ 

Jo 



e x dx — —Jit , 
2 



which can be used to obtain (3.100) by a simple change of variable. From [22] 
(p. 262) we have 



1.4 
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Figure 3.6 Plots of two Gaussian pulses, where g(x) in (3.101) for m = 0, with a 
1,0.1. 
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Theorem 3.5: Let lirtijc^oo x p f(x) — A. Then 

1. f^° f(x) dx converges if p > 1 and — oo < A < oo. 

2. / a °° /Or) fifx diverges if p < 1 and A ^ (A may be infinite). 

We see that lim^^oo* e x = (perhaps via l'Hopital's rule). So in The- 
orem 3.5 /(*) = e _ * , and p — 2, with A = 0, and so L e~ x dx converges. 
Define 



1m 



= / e~ x dx = / e- y 
Jo Jo 



<iv 



and let liiriM^oo I M — I. Then 



<iv 



= /" [ e-^+y 2 Uxdy 
Jr m J 

for which /?m is the square OASC in Fig. 3.7. This square has sides of length M. 
Since e - ^ +> ' ' > 0, we obtain 

f f e - (x2+y2) dx dy < I M < f f e - (x2+y2) dx dy, (3.104) 

JRi J JRij J 



M\/2 



M 



v v 



y 
l 

E 
- A 


i 


\ 6 


^ 




O 


( 


Z D 



Figure 3.7 Regions used to establish (3.103). 
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where Ry is the region in the first quadrant bounded by a circle of radius M. 
Similarly, Ry is the region in the first quadrant bounded by a circle of radius \p2M. 
Using polar coordinates, r 2 — x 2 + y 2 and dx dy — r dr d(f>, so (3.104) becomes 

nir/2 pM nir/2 rs/2M 

i I e~ r r dr d<p < 1 2 M < / e~ r rdrd(t>. (3.105) 

J0=O Jr=Q Jij>=Q Jr=a 

Since -j£e~ x2 = xe'* 2 we have f Q M re~ rl dr = -\[e~ x2 ]^ = ±[1 - e~ M \ 
Thus, (3.105) reduces to 

j[l-e- M2 l<I 2 M <^ll-e- 2M \ (3.106) 

If we now allow M -> oo in (3.106), then /^ -» j, implying that I 2 = j, or 
/ = jyfjz. This confirms (3.103). 

In probability theory it is quite important to be able to compute functions such 
as the error function 

2 f x 2 
ert{x)=—= / e~ f dt. (3.107) 

\/x Jo 

This has wide application in digital communications system analysis, for example. 
No closed-form 4 expression for (3.107) exists. We may therefore try to compute 
(3.107) using series expansions. In particular, we may try working with the Maclau- 
rin series expansion for e x : 



erf(x) 



^ — ^ A 

u=o 



2 r x 

jn Jo 

_ 2 y,(-l)« f 

~^h k\ Jo 



dt 



t 2k dt 



2 ~ (-l) k x 2k+1 

= -=> - — . (3.108) 

JWf^kW + 1) 

However, to arrive at this expression, we had to integrate an infinite series term 
by term. It is not obvious that we can do this. When is this justified? 
A power series is any series of the form 

CO 

f(x) = J^a k (x-x ) k . (3.109) 

k=0 

Clearly, Taylor and Maclaurin series are all examples of power series. We have the 
following theorem. 

A closed-form expression is simply a "nice" formula typically involving more familiar functions such 
as sines, cosines, tangents, polynomials, and exponential functions. 
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Theorem 3.6: Given the power series (3.109), there is an R > (which may 
be R = +oo) such that the series is absolutely convergent for \x — xq\ < R, and is 
divergent for \x — xq\ > R. At x — xo — R and at x — xq — — R, the series might 
converge or diverge. 

Series (3.109) is absolutely convergent if the series 

oo 
h(x) = J^\a k (x-x ) k \ (3.110) 

converges. We remark that absolutely convergent series are convergent. This means 
that if (3.110) converges, then (3.109) also converges. (However, the converse is 
not necessarily true.) We also have the following theorem. 

Theorem 3.7: If 

00 

f (x) — 2_\ a k{x — xq) for \x — xq\ < R, 

where R > is the radius of convergence of the power series, then f(x) is 
continuous and differentiable in the interval of convergence x e (xo — R, xq + R), 
and 

oo 

fW(x) = Y^ka k (x-x ) k -\ (3.111a) 

k=\ 

/ f(t)dt = J27^h( x - x o) k+1 - (3.111b) 

This series [Eq. (3.111a,b)] also has a radius of convergence R. 

As a consequence of Theorem 3.7, Eq. (3.108) is valid for — oo < x < oo (i.e., the 
radius of convergence is R — +oo). This is because the Maclaurin expansion for 

e x had R — +oo. 

Example 3.9 Here we will find an expression for the error involved in trun- 
cating the series for erf(x) in (3.108). 

From (3.71) for some f € [0, x] (interval endpoints may be included because of 
continuity of the function being approximated) 

n fc 

k=0 ' 
=Pn(x) 
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where 

e,.(x\ = 

(«+l)! 

Thus, where x — —t 2 , so for some £ such that — t 2 < § < 

<?"' 2 = p„(-t 2 ) + e n (-t 2 ), 

and hence 

2 f x 2 f x 

erf(x) = —= p n (-t 2 )dt + —= e n (-t 2 )dt, 

Jit Jo JTI Jo 



/0 \/n JO 

=9n (*) =e„ CO 

where the degree « polynomial 

2 r 
^„(x) = — — / 

V 77 Jo 
is the approximation, and we are interested in the error 

2 f x 
e„(x) = erf(x) - #„(*) = — — / e n {-t 2 )dt. 

Jn Jo 



A (-\) k t 2k 
^ k\ 

Lk=o 


dt - 


2 " { -lf x 2k+l 

' ^ho k ^ 2k + i ) 



Clearly 

e n (x) = ^=P^f X t 2n+2 eSdt, 

V^r (« + 1)! J 



where we recall that f depends on f in that — t 2 < % < 0. There is an integral 
mean-value theorem, which states that for /(?), g(0 S C[a, £>] (and g(f) does not 
change sign on the interval [a, b]) there is a f € [a, b] such that 



/" g(t)f(t)dt = f(0 f 

J a J a 



-b 

g(t)f(t)dt = f(0 I g {t)dt. 



Thus, there is a f € [— x , 0], giving 



2 ,(-l)" +1 x 2 "+ 3 
e n (x) = — =e 4 



V^r (n+l)!2n + 3' 

Naturally the error expression in Example 3.9 can be used to estimate how many 
terms one must keep in the series expansion (3.108) in order to compute erf(x) to 
a desired accuracy. 
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3.6 ASYMPTOTIC SERIES 

The Taylor series expansions of Section 3.5 might have a large radius of conver- 
gence, but practically speaking, if x is sufficiently far from xq, then many many 
terms may be needed in a computer implementation to converge to the correct 
solution with adequate accuracy. This is highly inefficient. Also, if many terms 
are to be retained, then rounding errors might accumulate and destroy the result. 
In other words, Taylor series approximations are really effective only for x suf- 
ficiently close to xq (i.e., "small x"). We therefore seek expansion methods that 
give good approximations for large values of the argument x. These are called the 
asymptotic expansions, or asymptotic series. This section is just a quick introduction 
based mainly on Section 19.15 in Kreyszig [24]. Another source of information on 
asymptotic expansions, although applied mainly to problems involving differential 
equations, appears in Lakin and Sanchez [25]. 

Asymptotic expansions may take on different forms. That is, there are different 
"varieties" of such expansions. (This is apparent in Ref. 25.) However, we will 
focus on the following definition. 

Definition 3.5: A series of the form 



k=0 



(3.112) 



for which q e R (real-valued constants), and x e R is called an asymptotic expan- 
sion, or asymptotic series, of a function f(x), which is defined for all sufficiently 
large x if, for every n e Z + 



/<*>- E 




as x -> oo, 



(3.113) 



and we shall then write 



/(*) ~ J2 



£1 



k=0 



It is to be emphasized that the series (3.112) need not converge for any x. The 
condition (3.113) suggests a possible method of finding sequence (ck)- Specifically 



[fix) 



f(x) - c -► or co = lim f(x), 



CO 



- \x -> or c\— lim [f(x) — co]x, 



[fix) 



CO 



Cl C2| 2 

2 ' X 



r] 



or C2 = lim 

X— >00 



[fix) -co Jx 2 , 
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or in general 



lim 

X — >oo 



n-\ 



/(*) - E 



C L 

■k 



X 
k=0 J 



(3.114) 



for n > 1. However, this recursive procedure is seldom practical for generating 
more than the first few series coefficients. Of course, in some cases this might 
be all that is needed. We remark that Definition 3.5 can be usefully extended 
according to 



f(x) ~ g(x) + h(x) 



E 

U=0 



fi 



for which 



fix) - g(x) 
h(x) 



k=0 



(3.115) 



(3.116) 



The single most generally useful method for getting (cjt) is probably to use "inte- 
gration by parts." This is illustrated with examples. 

Example 3.10 Recall erf(x) from (3.107). We would like to evaluate this func- 
tion for large x [whereas the series in (3.108) is better suited for small x; see the 
error expression in Example 3.9]. In this regard it is preferable to work with the 
complementary error function 



erfc(x) = 1 — erf(x) 



—— / e ' dt. 

s/7T Jx 



(3.117) 



We observe that erf(oo) = 1 [via (3.103)]. Now let r = t , so that dt 
With this change of variable 

1 f°° _i _ T 
erfc(x) = — — I x 2 e dr. 
Jtt J x 2 



U-i 



' 2 dx. 



(3.118) 



Now observe that via integration by parts, we have 






T-2e- T dr = -T- 1/2 e- T \°° 2 



2J X 2 



x 2e T dx 



= X -e~^ 

x 2 



i r 

2J X 2 



x-2e~ T dx = -x- 3/2 e- T \°° 2 



x 2 e T dx 
3 



/ 



x 2 e T dx 



f 

Jx 2 



X 2 e T dx, 
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and so on. We observe that this process of successive integration by parts has 
generated integrals of the form 



F „(*)= / 

Jx 2 



-(2n+l)/2 -T 



e Jt 



(3.119) 



for n e Z + , and we see that erfc(x) = -j=Fo(x). So, if we apply integration by 
parts to F n (x), then 



I T -(2« + D/2 e -r rfr 

*2 



. T -(2»+l)/2 e -r|00 _ ^L±l p T -(2„+3)/2 e -T ^ 
* 2 J v 2 



-(2n + l)-x 2 _ 2n + 1 



/; 



-v^i\- dt 



so that we have the recursive expression 



F n (x) = 



2m+ 1 



-2n+l 



-F n+ i(x) 



(3.120) 



which holds for n e Z + . This may be rewritten as 



e x F n (x) 



2« + 1 



r 2n+l 



-e* F B+ i(*). 



(3.121) 



Repeated application of (3.121) yields 



e x2 F (x) = - - J/f,W, 
x 2 

,2 1 1 1-3,2 

e* F (*) = - - — + — e * F 2 (x), 
x 2 1 1 1-3 1 • 3 • 5 x 2 



and so finally 



e x F (x) = 



1 1 1-3 

x 2x 3 2 2 x 5 



+ (-D 



n _i 1 ■ 3 ■ ■ ■ (2« - 3) 



2« — 1 v2n — 1 



=5 2 „_i(x) 

„ 1 • 3 • • • (2n - 1) r 2 
+ (-D" ^ -«* F n (x). 



(3.122) 
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From this it appears that our asymptotic expansion is 

r 2 1 1 1-3 „ , 1 • 3 • • • (2m - 3) 

e F ° W ~x-2^ + 2^--- + ( - 1) 2-1^-1 +■- ^ 123 ) 

However, this requires confirmation. Define K n — (— 2)~"[1 • 3 • • • (2m — 1)]. From 
(3.122), we have 

[e x2 F (x) - S 2 „-i(x)]x 2n - 1 = K n e x \ 2n - x F n (x). (3.124) 

We wish to show that for any fixed n — 1, 2, 3, . . . the expression in (3.124) on 
the right of the equality goes to zero asi -> oo. In (3.119) we have 









1 


1 








< 










r (2« + l)/2 - 


y2n+l 


for all t 


> x 2 , 


which 


gives the bound 








/>OC 


e~ T 1 


/•CO 




F„(x) 


- / 


dx < 


/ e 




L* 


r (2n+l)/2 - x 2n+l 


U 


But this 


implies that 







^ r ^ = ^rr- (3- 125 ) 



2 

\K n \e x \ 2 "- l F n ( x ) < \K n \e x \ 2n - 1 *"" '*"' 



v-2n+l y2 

I K" I 

and L -j i — > for x — > oo. Thus, immediately, (3.123) is indeed the asymptotic 

2 

expansion for e* Fq(x). Hence 



1 -f 
erfc(x) ~ — —e 



1 1 1-3 „ , 1 • 3 • • • (2m - 3) 

1 L (-1)"" 1 + ■ ■ 

x 2x 3 2 2 x 5 + l ' 2"- 1 x 2 "- 1 + 

(3.126) 



We recall from Section 3.4 that the integral L ^SI dt was important in analyzing 
the Gibbs phenomenon in Fourier series expansions. We now consider asymptotic 
approximations to this integral. 

Example 3.11 The sine integral is 

f x sint 

~ Jo t 

and the complementary sine integral is 

f 00 sin? 

si(x) = / dt. (3.128) 

Jx t 



Si(x) = / dt, (3.127) 

o 
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It turns out that si(0) = j, which is shown on p. 277 of Spiegel [22]. We wish to 
find an asymptotic series for Si(x). Since 

n r°° sint f x sint f°° sint 

-=/ dt = / df+/ Jf = Si(x) + si(x), (3.129) 

2 Jo t J t Jx t 

we will consider the expansion of si(x). If we integrate by parts in succession, then 

1 



/; 

r 

J X 

f 

J X 



i ' f°° ! 

t sint dt — — cosx — 1 • / — 

x J x V 

,00 , 

Jx 7* 



cost dt, 



-7 1 

£ cos t dt — -sinx + 2 



1 



f sinf df = — 7 cosx — 3 • 

x i 



1 



■ sin? df, 
cost dt, 

■ sin t dt, 

Jx ' 

and so on. If n € N, then we may define 

,00 ,00 

,y n (x) = / t~ n sin tdt, c„(x) — I t~ n cost dt. 

J X J X 



poo 

I t cos tdt — jsinx + 4- 

Jx x 4 



,oo l 
Jx ^ 



Therefore, for odd n 



and for even n 



1 

Sn 00 = — - COS X - n C n +\(x), 

x" 



c n {x) = -sinx +n s n+ \(x). 

x" 



(3.130) 



(3.131a) 



(3.131b) 



We observe that Si(x) = si(x). Repeated application of the recursions (3.131a,b) 
results in 

11 11. 
s\(x) — cos* H 7t- sinx — 1 • 2st,(x) 



x 
1- 1 



1 • 1 



1 -2 



1-2-3 



■ COS X 



sinx + 1-2-3 -4s 5 (x), 



or in general 

s\(x) = cosx 



11 1-2 
x x 3 

1-1 1-2-3 



x x* 

+ (-l)"(2»)Lv 2 „ +1 (x) 



(-D 



„+i(2«-2)! 



r 2/i — 1 



+ ••• + (-1) 



+1 (2n-l)! 

v 2n 



(3.132) 
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for n e N (with 0! = 1). From this, we obtain 

[si(x) - S 2n (x)]x 2n = (-l) n (2n)\x 2n s 2n+1 (x) 



(3.133) 



for which S2n(x) is appropriately defined as those terms in (3.132) involving sinx 
and cosx. It is unclear whether 



lim x 2n S2n+i(x) — 
x— >oo 



(3.134) 



for any n e N. But if we accept (3.134), then 
si(x) ~ S2n(x) = cosx 



1-1 1-2 „, ,(2/i-2)! 



sinx 



id_i4^ + ... +( _ ir+ .< 2 »-') ! 



r ln 



(3.135) 



Figure 3.8 shows plots of Si(x) and the asymptotic approximation j — S2n(x) for 
n = 1, 2, 3. We observe that the approximations are good for "large x," but poor 
for "small x," as we would expect. Moreover, for small x, the approximations are 
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^ 1 



E 
< 
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Figure 3.8 Plot of Si(x) using MATLAB routine sinint, and plots of 5- — 52„(x) for n ■■ 
1,2,3. The horizontal solid line is at height n /2. 
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better for smaller n. This is also reasonable. Our plots therefore constitute informal 
verification of the correctness of (3.135), in spite of potential doubts about (3.134). 

Another approach to finding (q.) is based on the fact that many special func- 
tions are solutions to particular differential equations. If the originating differential 
equation is known, it may be used to generate sequence (c*)- However, this section 
was intended to be relatively brief, and so this method is omitted. The interested 
reader may see Kreyszig [24]. 

3.7 MORE ON THE DIRICHLET KERNEL 

In Section 3.4 the Dirichlet kernel was introduced in order to analyze the manner 
in which Fourier series converge in the vicinity of a discontinuity of a 2n -periodic 
function. However, this was only done with respect to the special case of g(t) in 
(3.19). In this section we consider the Dirichlet kernel D n (t) in a more general 
manner. 

We begin by recalling the complex Fourier series expansion from Chapter 1 
(Section 1.3.3). If f(t) e L 2 (0, 2jt), then 

00 

fa) = Y, f » eint > (3 ' 136) 

n — — oo 

where /„ = (/, e n ), with e n (t) — e' nt , and 

(x,y) = ±- ^ x{t)y*(t)dt. (3.137) 

2n Jo 

An approximation to f(t) is the truncated Fourier series expansion 

L 

h(t)= J2 fne' nt e L 2 (0,2jt). (3.138) 



The approximation error is 

6i (0 = fit) - hit) e L 2 (0, 2it). (3.139) 

We seek a general expression for ei(t) that is hopefully more informative than 
(3.139). The overall goal is to generalize the error analysis approach seen in 
Section 3.4. Therefore, consider 



6l(0 = /(0- Y (f> e n)e„(t) 
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[via /„ = {/, e n ), and (3.138) into (3.139)]. Thus 

e £ (0 = /(0- M^/ f(x)e- jnx dx\e n (t) 



fit) , 



^/^/wl i>'" ( ' _J0 U* (3 - 140) 



Since 



z. 



^ e jn(t-x) = j + 2 ^ cos[ „( f _ x )] (3.141) 

n=—L n=\ 

(show this as an exercise), via (3.24), we obtain 

J^ sin[(L + \){t - *)] 

l+2> cos[«(f-x)] = j-? =27tD L (t-x). (3.142) 

^ sin[±(f-x)] 

Immediately, we see that 

ei(0 = /(0-/ f(x)D L (t-x)dx, (3.143) 

Jo 



where also (recall (3.139)) 



Jo 



fL(t)= I f(x)D L (t-x)dx. (3.144) 



Equation (3.144) is an alternative integral form of the approximation to /(f) origi- 
nally specified in (3.138). The integral in (3.144) is really an example of something 
called a convolution integral. The following example will demonstrate how we 
might apply (3.143). 

Example 3.12 Suppose that 

sinf, < t < n 



^ (f) "0, it < t < In " 

Note that f(t) is continuous for all t, but that f^'{t) = df(t)/dt is not continuous 
everywhere. For example, f^'{t) is not continuous at t — jr. Plots of fi(t) for 
various L (see Fig. 3.10) suggest that /l(?) converges most slowly to f(t) near 
t — it. Can we say something about the rate of convergence? 
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Therefore, consider 



M 



1 r 

2n J 



sinx 



sin[(L + \){7t - x)] 

sin[i(7r — x)] 



dx. 



(3.145) 



Now 



1 



-(JT — X) 



l ( — ) cos f —x 1 — cos ( — ) sin I -. 

\2/ \2 / V2/ \2 



COS —X 

.2 



and since sinx = 2sin(Ax) cos(lx) so (3.145) reduces to 



fL(7t) =Uo sin G x ) 


sin 


!H) ( 


T — X) 


dx 






i r*i 

= — / { cos 

In Jo I 


(L + l)x - ( L + - ) it 


— cos 


Lx — 


(l + \)* 


1 


sin 


(L + l)x - (l + it 


-\7l 

- 




2it(L+ 1) _ 




1 

2jtL 


sin 


Lx — I L H — 1 7r 

V 27 . 


- 7T 

- 




1 

2jr(L+ 1) . 


1 + sin I L H — 1 7T 

V 2; . 


1 

2jtL 


- 1 + sin 


[ L+l 2)\ 


1 2L + 1- (-1) L 





lit L(L + 1) 
We see that, as expected 



dx 



(3.146) 



lim A(tt) = 0. 

L—>-oo 

Also, (3.146) gives €/.(7r) = — /l(7t), and this is the exact value for the approxi- 
mation error at t — jt for all L. Furthermore 

1 

ki(7r)| oc - 

for large L, and so we have a measure of the rate of convergence of the error, at 
least at the point t = n. (Symbol "oc" means "proportional to.") 

We remark that f(t) in Example 3.12 may be regarded as the voltage drop 
across the resistor R in Fig. 3.9. The circuit in Fig. 3.9 is a simple half-wave 
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Ideal diode 




Figure 3.9 An electronic circuit interpretation for Example 3.12; here, v(t) = sin(f) for 
all t e R. 




Figure 3.10 A plot of f(t) and Fourier series approximations to f(t) [i.e., //,(?) for 
L = 3, 10]. 



rectifier circuit. The reader ought to verify as an exercise that 

L 

sin t 4- 

jt 2 



1 1 " l + (-l) n 

f L (t)= — + — sin t + > — — cos nt. 



n=2 



7T(l-n 2 ) 



(3.147) 



Plots of /i(f) for L — 3, 10 versus the plot of f(t) appear in Fig. 3.10. 
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3.8 FINAL REMARKS 

We have seen that sequences and series might converge "mathematically" yet not 
"numerically." Essentially, we have seen three categories of difficulty: 

1. Pointwise convergence of series leading to irreducible errors in certain regions 
of the approximation, such as the Gibbs phenomenon in Fourier expansions, 
which arises in the vicinity of discontinuities in the function being approxi- 
mated 

2. The destructive effect of rounding errors as illustrated by the catastrophic 
convergence of series 

3. Slow convergence such as illustrated by the problem of computing it with 
the Maclaurin expansion for tan -1 x 

We have seen that some ingenuity may be needed to overcome obstacles such 
as these. For example, a divide-and-conquer approach helped in the problem of 
computing n . In the case of catastrophic convergence, the problem was solved by 
changing the computational algorithm. The problem of overcoming Gibbs overshoot 
is not considered here. However, it involves seeking uniformly convergent series 
approximations. 



APPENDIX 3.A CO ORDINATE R OTATATION DI GITAL 
COMPUTING (CORDIC) 

3.A.1 Introduction 

This appendix presents the basics of a method for computing "elementary func- 
tions," which includes the problem of rotating vectors in the plane, computing 
tan _1 x, sinS, and co&6. The method to be used is called "coordinate rotation 
digital computing" (CORDIC), and was invented by Jack Voider [3] in the late 
1950s. However, in spite of the age of the method, it is still important. The method 
is one of those great ideas that is able to survive despite technological changes. It 
is a good example of how a clever mathematical idea, if anything, becomes less 
obsolete with the passage of time. 

The CORDIC method was, and is, desirable because it reduces the problem of 
computing apparently complicated functions, such as trig functions, to a succession 
of simple operations. Specifically, these simple operations are shifting and adding. 
In the 1950s it was a major achievement just to build systems that could add two 
numbers together because all that was available for use was vacuum tubes, and to a 
lesser degree, discrete transistors. However, even with the enormous improvements 
in computing technology that have occurred since then, it is still important to reduce 
complicated operations to simple ones. Thus, the CORDIC method has survived 
very well. For example, in 1980 [5] a special-purpose CORDIC VLSI (very large 
scale integration) chip was presented. More recent references to the method will 
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be given later. Nowadays, the CORDIC method is more likely to be implemented 
with gate-array technology. 

Since the CORDIC method involves the operations of shifting and adding only, 
once the mathematics of the method is understood, it is easy to build CORDIC 
computing hardware using the logic design methods considered in typical elemen- 
tary digital logic courses or books. 5 Consideration of CORDIC computing also 
makes a connection between elementary mathematics courses (calculus and linear 
algebra) and computer hardware systems design, as well as the subject of numerical 
analysis. 

3.A.2 The Concept of a Discrete Basis 

The original paper of Voider [3] does not give a rigorous treatment of the mathe- 
matics of the CORDIC method. However, in this section (based closely on Schelin 
[6]), we will begin the process of deriving the CORDIC method in a mathemati- 
cally rigorous manner. The central idea is to represent operands (e.g, the 9 in sinf9) 
in terms of a discrete basis. When the discrete basis representation is combined 
with appropriate mathematical identities the CORDIC algorithm results. 

Let R denote the set of real numbers. Everything we do here revolves around 
the following theorem (From Schelin [6]). 

Theorem 3.A.1: Suppose that $ k e R f or k e {0, 1, 2, 3, . . .} satisfy 9 > B\ > 
6» 2 > • • • > 6„ > 0, and that 

n 

Ok < Y^ 9 J +0 " for ° - k - "' (3.A.la) 



and suppose 9 e R satisfies 



\6\<J2 9 i- ( 3 - A - lb ) 



If 6 (0) = 0, and 9 (k+l) = 9 {k) + S k 9 k for < k < n, where 

1, if9>9 (k) 



then 



«*H -i iie <9^ (3A ' lc) 



\9 - 9 (k) \ < J2 e .i + e n for < k < n, (3.A.ld) 

j=k 



and so in particular \9 — 0^" +1 '| < 9 n . 



The method is also easy to implement in an assembly language (or other low-level) programming 
environment. 
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Proof We may use proof by mathematical induction 6 on the index k. For k — 

n n 

\8-6 (O) \ = \6\<J2 J <J2 e J +0 n 

.7=0 j=0 

via (3.A.lb). 

Assume that \9 - 8 (k) \ < T,"=k e i + 9 " is tme ' and consider \9 - 9 (k+l) \. Via 
(3.A.lc), S k and 9 — 0® have the same sign, and so 



|0-0 ( * +1) | 



)W-8 k $ k \ = \\ 



j(*)i 



ft|. 



Now, via the inductive hypothesis (i.e., \6 — 9^ >\ < Yl']=k ®i + ^n)> we have 



}(*)! 



Via (3. A. la), we obtain 



j=k 



j=k+l 



(3.A.2a) 



E^ 

;'=*+! 



<-0* 



so that 

n 

J2 °i + e " 
j=k+i 

Combining (3.A.2a) with (3. A. 2b) gives 



< \9 -8 (k) \ -9 k . 



(3.A.2b) 



\e-9 {k+l) \ = \\9-9 {k) \-8 k \< J2 e i+ e n 

i=k+\ 

so that (3 A. Id) holds for k replaced by k + 1, and so (3 A. Id) holds via induction. 

We will call this result Schelin's theorem. The set {9 k } is called a discrete basis 
if it satisfies the restrictions given in the theorem. 

In what follows we will interpret 9 as an angle. However, note that 9 in this 
theorem could be more general than this. Now suppose that we define 



9 k = tan" 1 2~ k 



(3.A.3) 



for k = 0, 1, . . . , n. Table 3.A.1 shows typical values for 9 k as given by (3.A.3). 
We see that good approximations to 9 can be obtained from relatively small n 
(because from Schelin's theorem \9 — 9 ( - n+1) \ < 9 n ). 

A brief introduction to proof by mathematical induction appears in Appendix 3.B. 
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TABLE 3.A.1 Values 
for Some Elements of 
the Discrete Basis given 
by Eq. (3.A.3) 



k 


Ok 


= tan _1 2-' : 







45° 


1 




26.565° 


2 




14.036° 


3 




7.1250° 


4 




3.5763° 



Clearly, these satisfy 0q > 0\ > ■ ■ ■ > 6 n > 0, which is one of the restrictions in 
Schelin's theorem. 

The mean-value theorem of elementary calculus says that there exists a £ e 
[xq, xq + A] such that 



df(x) 



dx 



*=? 



f(x + A) - f(x ) 

A 



so if f(x) — tan x, then df(x)/dx — 1/(1 + x ), and thus 



1 



1+x 2 



x£[2-< k + l \2- k ] 



Qk - e k+ i 

2-k _ 2-(*+i) 



fc+i 



1 



^. =2 -(*+l) 



- 2 _* _ 2 _ (t+ i) - 1+JC 2 

because the slope of tan -1 x is largest at x — 2~ < - k+l \ and this in turn implies 
2~k _ 9-(*+i) 2* +2 — 2' t+1 2* +1 



61* - ^+i < 



and 



1 + 2- 2 ^+D 1 + 2 2 ^ +1 ' 1 + 2 2 ^ +1 > 

2 k 
k > 



\+2 2k 



because tan l x > x -£- tan ! x = -jj^iC* > 0), for k = 0, 1, . . . , n (let x — 2 k ). 



k\l 



'For x > 



rf x 1 — a: 



1+x 2 dx 1+x 2 (1+x 2 ) 2 



f* 1 x 

/ T dt > 



-=- => tan x > r- 

: 2 1+x 2 
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Now, as a consequence of these results 

Ok ~ n = (0 k - ftfc+i) + (0 k+l - k+2 ) + ■■■ + ($„-2 ~ n -!) + (9 n -X - n ) 
n-\ 



v^ 2j+l ( _ 2/+1 A 

2^ ! + 2 2(./+D \ i °i+ l ~ i + 220+D J 



< 



2-' 



^ l+2 2 i 



./'=*+ 1 



j=k+\ 

implying that 






0* < J^ Oj + e„ 

y=*+l 



for A: = 0, 1, ...,«, and thus (3.A.la) holds for {6>yt} in (3.A.3), and so (3.A.3) is 
a concrete example of a discrete basis. If you have a pocket calculator handy then 
it is easy to verify that 

3 

J2 j = 92.73° > 90°, 
;=o 

so we will work only with angles 6 that satisfy \6\ < 90°. Thus, for such 0, there 
exists a sequence {S^} such that S^ e {—1, +1}, where 

= £**& + e„+i = ( " +1) + e„+i, (3.A.4) 

where |e n +i| < 0„ — tan -1 2~". Equation (3.A.lc) gives us a way to find {Sk}- We 
can also write that any angle 6 satisfying \0\ < ^ (radians) can be represented 
exactly as 

oo 

= J2 S k°k 

k=0 

for appropriately chosen coordinates {8k}. 

In the next section we will begin to see how useful it is to be able to represent 
angles in terms of the discrete basis given by (3. A. 3). 



TLFeBOOK 



112 SEQUENCES AND SERIES 



3.A.3 Rotating Vectors in the Plane 



No one would disagree that a basic computational problem in electrical and com- 
puter engineering is to find 



e J z, 



€R, 



(3.A.5) 



where j — V— 1, z — x + jy, and z' = x' + jy' (x, y, x' ', y' e R). Certainly, this 
problem arises in the phasor analysis of circuits, in computer graphics (to rotate 
objects, for example), and it also arises in digital signal processing (DSP) a lot. 8 

Thus, z and z' are complex variables. Expressing them in terms of their real and 
imaginary parts, we may rewrite (3.A.5) as 

x + jy' — (cos 9 + j sin9)(x + jy) — [x cos 9 — y sin 9] + j[x sin 6* + y cos 6], 

and this can be further rewritten as the matrix-vector product 

(3.A.6) 



cost 
sinf 



- sin 9 
cos 6 



which we recognize as the formula for rotating the vector [xy] T in the plane to 

y'f. 

Recall the trigonometric identities 



[x'y'f 



sin 6 — 



tan 9 



VT 



COS 9 — 



tan z 



Vl + tan 2 #' 



(3.A.7) 



and so for 9 — 9^ in (3. A. 3) 



sin0£ = 



Vl+2- 2 *' 



cos 9k = 



VI +2~ 2k ' 



(3.A.: 



Thus, to rotate Xk + jyic by angle Sj i 9/ C to x^+i + jyk+i is accomplished via 



Xk+l 

yk+i 



^\ + 2~ 2k L S k 2 



S k 2~ k 

1 



Xk 

yk 



(3.A.9) 



In DSP it is often necessary to compute the discrete Fourier transform (DFT), which is an approximation 
to the Fourier transform (FT) and is defined by 



E( 2izkn \ 
expl-J-jy-**): 



where {jtj.} is the samples of some analog signal, i.e., x^ = x(kT), where k is an integer and T is a 
positive constant; x(t) for t s R is the analog signal. Note that additional information about the DFT 
appeared in Section 1.4. 
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where we've used (3. A. 6). From Schelin's theorem, < -"+ 1 ' 1 ss y, so if we wish to 
rotate x + jy = x + jy by y ( " +1) to x„+i + jy„+i (« x' + jy') then via (3.A.9) 



" +1 = I FT 




S\2~ l 



(3.A.10) 



Define 



*„=n 



k=0 



vT 



-2k 



(3.A.11) 



where Ojfc=0 "* = «n«n-l«n-2 • • • «1«0- 
Consider 



X n + 1 
9n+l 



1 

5 W 2" 



-<5„2 

1 



5,2" 



-Si2~ 

1 



1 
So 



-S 

1 



vo 



(3A.12) 
which is the same expression as (3. A. 10) except that we have dropped the multi- 
plication by K n . We observe that to implement (3 .A. 12) requires only the simple 
(to implement in digital hardware, or assembly language) operations of shifting 
and adding. Implementing (3 A. 12) rotates x + jy by approximately the desired 
amount, but gives a solution vector that is a factor of l/K n longer than it should 
be. Of course, this can be corrected by multiplying [x n +\ y n +\] T by K n if desired. 
Note that in some applications, this would not be necessary. 
Note that 



cosf 
sin0 



- sin0 
cosy 



K„ 



1 
«„2-" 

1 

5.2" 1 



-*„2-" 

1 



5i2" 

1 



l 



1 
So 



So 



(3.A.13) 



and so the matrix product in (3. A. 12) [or (3. A. 10)] represents an efficient approxi- 
mate factorization of the rotation operator in (3. A. 6). The approximation gets better 
and better as n increases, and in the limit as n -> oo becomes exact. 

The computational complexity of the CORDIC rotation algorithm may be des- 
cribed as follows. In Eq. (3 A. 10) there are exactly 2n shifts (i.e., multiplications 
by 2~ k ), and 2« + 2 additions, plus two scalings by factor K n . As well, only n + \ 
bits of storage are needed to save the sequence {8^}. 

We conclude this section with a numerical example to show how to obtain the 
sequence {S^} via (3.A.lc). 
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Example 3.A.1 Suppose that we want to rotate a vector by an angle of 9 = 
20°, and we decide that n — A gives sufficient accuracy for the application at hand. 
Via (3.A.lc) and Table 3.A.1, we have 

(1) =$ = 45° as <5 = +1 since 9 > (9 (0) = 0° 

0(2) = 0(1) +Sl 0j 

= 45° - 26.565° as Si = -1 since 9 < 9 (1) 
= 18.435° 

0(3) = 0(2) + g 2d2 

= 18.435° + 14.036° as <5 2 = +1 since 9 > 9 (2) 
= 32.471° 
0^ = 9& + s 303 

= 32.471° - 7.1250° as S 3 = -1 since < 9 (3) 
= 25.346° 
0< 5 ) = 0< 4 > + 5 4 4 

= 25.346° - 3.5763° as S 4 = -1 since < (9 (4) 
= 21.770° «6» 

Via (3.A.4) and \9 - 6» (n+1) | < 6„ 

\e 5 \ = 1.770° < 04 = 3.5763° 

so the error bound in Schelin's theorem is actually somewhat conservative, at least 
in this special case. 

3.A.4 Computing Arctangents 

The results in Section 3. A. 3 can be modified to obtain a CORDIC algorithm for 
computing an approximation to 6 — tan _1 (y/x). The idea is to find the sequence 
{Sk\k = 0, 1, . . . , n — 1, n] to rotate the vector [ x y ] T — [ xq yo ] T to the 
vector [x n y n ] T , where y n « 0. More specifically, we would select 8k so that 

Iw+il < \yk\- 

Let 9 denote the approximation to 6. The desired algorithm to compute 9 may 
be expressed as Pascal-like pseudocode: 

6 :=0; 

x :=x;y := y; 
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for k := to n d 


d begin 


ify/<> 


Other 


begin 


**:= 


= -1; 




end 






else begin 




* k := 


= +1; 




end 


; 




§:=- 


S k d k + 


9; 


Xfr+1 : 


= x k - 


W k Yk\ 


y^+i : 


= S k 2~ 


k Xk+Yk\ 


end ; 







In this pseudocode 



Xk+l 




l 


-S k 2~ k ' 


Xk 


yk+\ 




. S k 2~ k 


1 


. yk 



and we see that for the manner in which sequence {8k} is constructed by the 
pseudocode, the inequality \y k +i\ < \yk\ is satisfied. We choose n to achieve the 
desired accuracy of our estimate 6 of 6, specifically, \6 — 6\ <9 n . 

3.A.5 Final Remarks 

As an exercise, the reader should modify the previous results to determine a 
CORDIC method for computing cos0, and sinf?. [Hint: Take a good look at 
(3.A.13).] 

The CORDIC philosophy can be extended to the computation of hyperbolic 
trigonometric functions, logarithms 9 and other functions [4, 7]. It can also perform 
multiplication and division (see Table on p. 324 of Schelin [6]). As shown by Hu 
and Naganathan [9], the rate of convergence of the CORDIC method can be accel- 
erated by a method similar to the Booth algorithm (see pp. 287-289 of Hamacher 
et al. [10]) for multiplication. However, this is at the expense of somewhat more 
complicated hardware structures. A roundoff error analysis of the CORDIC method 
has been performed by Hu [8]. We do not present these results in this book as they 
are quite involved. Hu claims to have fairly tight bounds on the errors, however. 
Fixed-point and floating-point schemes are both analyzed. A tutorial presentation 
of CORDIC-based VLSI architectures for digital signal processing applications 
appears in Hu [11]. Other papers on the CORDIC method are those by Timmer- 
mann et al. [12] and Lee and Lang [13] (which appeared in the IEEE Transactions 
on Computers, "Special Issue on Computer Arithmetic" of August 1992). An alter- 
native summary of the CORDIC method may be found in Hwang [14]. Many of 
the ideas in Hu's paper [11] are applicable in a gate-array technology environ- 
ment. Applications include the computation of discrete transforms (e.g., the DFT), 
digital filtering, adaptive filtering, Kalman filtering, the solution of special linear 

y A clever alternative to the CORDIC approach for log calculations appears in Lo and Chen [15], and 
a method of computing square roots without division appears in Mikami et al. [16]. 
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systems of equations (e.g., Toeplitz), deconvolution, and eigenvalue and singular 
value decompositions. 



APPENDIX 3.B MATHEMATICAL INDUCTION 

The basic idea of mathematical induction is as follows. Assume that we are given 
a sequence of statements 

So, S\, ... , S„, . . . 

and each Si is true or it is false. To prove that all of the statements are true (i.e., 
to prove that S n is true for all n) by induction: (1) prove that S n is true for n — 0, 
and then (2) assume that S n is true for any n — k and then show that S n is true for 
n = k+\. 

Example 3.B.1 Prove 

" . 2 »(»+l)(2»+l) 

S n = 2_^ 1 = 7 ' ">0. 

i=0 

Proof We will use induction, but note that there are other methods (e.g., via z 
transforms). For n — 0, we obtain 

c V^-2 n a »(»+l)(2» + l) , 

So = 2^ ' = ° and 7 1«=0 = 0. 

(=0 

Thus, S n is certainly true for n — 0. 

Assume now that 5„ is true for n — k so that 

J t = E' 2 =* ( * + T +1) - (3-B.l) 



,=o 



We have 



and so 



5 ft+ i = j^ i 2 = J^ i 2 + (k+ l) 2 = S k + (k + l) 2 

i=0 i=0 



S k + (k+ \y = + (A: + l) z = 

6 6 

_ n(n+ 1)(2« + 1) 

— ~ \n=k+l 

6 
where we have used (3.B.1). 
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Therefore, S„ is true for n — k + 1 if S n is true for n — k. Therefore, S n is true 
for all n > by induction. 

APPENDIX 3.C CATASTROPHIC CANCELLATION 

The phenomenon of catastrophic cancellation is illustrated in the following out- 
put from a MATLAB implementation that ran on a Sun Microsystems Ultra 10 
workstation using MATLAB version 6.0.0.88, release 12, in an attempt to compute 
exp(— 20) using the Maclaurin series for exp(x) directly: 



term k x~k/ k ! 

1.000000000000000 

1 -20.000000000000000 

2 200.000000000000000 

3 -1333.333333333333258 

4 6666.666666666666970 

5 -26666.666666666667879 

6 88888.888888888890506 

7 -253968.253968253964558 

8 634920.634920634911396 

9 -1410934.744268077658489 

10 2821869.488536155316979 

11 -5130671 .797338464297354 

12 8551119.662230772897601 

13 -13155568.711124267429113 

14 18793669.587320379912853 

15 -25058226.116427175700665 

16 31322782.645533967763186 

17 -36850332.524157606065273 

18 40944813.915730677545071 

19 -43099804.121821768581867 

20 43099804.121821768581867 

21 -41047432.496973112225533 

22 37315847.724521011114120 

23 -32448563.238713916391134 

24 27040469.365594934672117 

25 -21632375.492475949227810 

26 16640288.840366113930941 

27 -12326139.881752677261829 

28 8804385.629823340103030 

29 -6071990.089533339254558 

30 4047993.393022226169705 

31 -2611608.640659500379115 

32 1632255.400412187911570 

33 - 989245 . 69721 9507652335 

34 581909.233658534009010 

35 -332519.562090590829030 

36 184733.090050328260986 

37 -99855.724351528784609 

38 52555.644395541465201 
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39 -26951.612510534087050 

40 13475.806255267041706 

41 -6573.564026959533294 

42 3130.268584266443668 

43 -1455.938876402997266 

44 661 .790398364998850 

45 -294.129065939999407 

46 127.882202582608457 

47 - 54 . 41 7958545790832 

48 22.674149394079514 

49 - 9 . 254754854726333 

50 3.701901941890533 

51 -1.451726251721777 

52 0.558356250662222 

53 -0.210700471948008 

54 0.078037211832596 

55 -0.028377167939126 

56 0.010134702835402 

57 - . 003556036082597 

58 0.001226219338827 

59 -0.000415667572484 

60 0.000138555857495 

61 -0.000045428149998 

62 0.000014654241935 

63 -0.000004652140297 

64 0.000001453793843 

65 -0.000000447321182 

66 0.000000135551873 

67 -0.000000040463246 

68 0.000000011900955 

69 -0.000000003449552 

70 0.000000000985586 

71 -0.000000000277630 

72 0.000000000077119 

73 -0.000000000021129 

74 0.000000000005710 

75 -0.000000000001523 

76 0.000000000000401 

77 -0.000000000000104 

78 0.000000000000027 

79 -0.000000000000007 

80 0.000000000000002 

81 -0.000000000000000 

82 0.000000000000000 

83 -0.000000000000000 

84 0.000000000000000 

85 -0.000000000000000 

86 0.000000000000000 

87 -0.000000000000000 

88 0.000000000000000 

exp(-20) from sum of the above terms = 0.000000004173637 
True value of exp(-20) = 0.000000002061154 
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PROBLEMS 

3.1. Prove the following theorem: Every convergent sequence in a metric space 
is a Cauchy sequence. 

3.2. Let /„(*) = x n for n e {1, 2, 3, . . .} = N, and /„(*) € C[0, 1] for all n e N. 

(a) What is f(x) = lim,,-,*, /„(*) (for x e [0, 1])? 

(b) Is f(x) e C[0, 1]? 

3.3. Sequence ix n ) is defined to be x n = (n + l)/(« + 2) for « € Z + . Clearly, if 
X = [0, 1) c R, then x n e X for all n e Z + . Assume the metric for metric 
space X is dix, y) — \x — y\ (x, y e X). 

(a) What is x — lirrin^oo x n l 

(b) Is X a complete space? 

(c) Prove that (x„) is Cauchy. 

3.4. Recall Section 3. A. 3 wherein the rotation operator was defined [Eq. (3. A. 6)]. 
(a) Find an expression for angle 9 such that for y ^ 



cos 9 
sin 9 



- sin 9 
cos 9 



x 

y 



X 





= G(9) 



where x' is some arbitrary nonzero constant. 

(b) Prove that G~ l i9) — G T id) [i.e., the inverse of G(9) is given by its 
transpose]. 

(c) Consider the matrix 

4 2 
A= 14 1 

_ 2 4 

Let 0„ xm denote an array (matrix) of zeros with n rows, and m columns. 
Find G(6»i), and Gi9 2 ) so that 



1 

2 xl 



0lx2 

Gi9 2 ) 



Gi$i) 2xl 

0lx2 1 



A = R, 



= Q< 



where R is an upper triangular matrix (defined in Section 4.5, if you do 
not recall what this is). 
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(d) Find Q~ T (inverse of Q T ). 

{Comment: The procedure illustrated by this problem is important in various 
applications such as solving least-squares approximations, and in finding the 
eigenvalues and eigenvectors of matrices.) 

3.5. Review Appendix 3. A. Suppose that we wish to rotate a vector [xy] T e R 2 
through an angle 9 — 25° ± 1°. Find n, and the required delta sequence (8k) 
to achieve this accuracy. 

3.6. Review Appendix 3. A. A certain CORDIC routine has the following pseu- 
docode description: 

Input x and z (|z| < 1); 
y :=0;z :=z; 
for k :— to n — 1 do begin 
if z k < then begin 

end 
else begin 

end; 
y k+ i :=y k + B k x2- k ; 

z k+1 :=z k -S k 2- k ; 

end; 

The algorithm' s output is y n . What is y„ ? 

3.7. Suppose that 

t 

x n (t) = 



n + t 



for n e Z + , and t e (0, 1) C R. Show that (x n (t)) is uniformly convergent 
on S = (0, 1). 

3.8. Suppose that u n > 0, and also that ^2T=o M « converges. Prove that ]~[^lo(l + 
u n ) converges. 

[Hint: Recall Lemma 2.1 (of Chapter 2).] 

3.9. Prove that limn^^x" = if \x\ < 1. 
3.10. Prove that for a, b, x e R 

1 1 1 



b\ ~ \ + \a-x\\ + \b 



(Comment: This inequality often appears in the context of convergence proofs 
for certain series expansions.) 
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3.11. Consider the function 



In N 



K N (x) = j—}^D n (x) 



n=0 

[recall (3.24)]. Since D n (x) is 2it -periodic, K^(x) is also 2jt -periodic. We 
may assume that x e [—n, n], 

(a) Prove that 

1 1 - cos(N + \)x 

K N (x) = — — — i '-. 

N + 1 1 — COS* 

[Hint: Consider ZLo sin ((« + 2>*) = Im [E*=o e^ n+ ^ x ] .] 

(b) Prove that 

K N (x) > 0. 



(c) Prove that 

1 /•* 

— / Kn(x) dx — 1. 

2tt 



J — 71 



[Comment: The partial sums of the complex Fourier series expansion of the 
2tt -periodic function f(x) (again x e [—it, jt]) are given by 



N 

'nx 



/n(x) = ^2 f^ 



n=-N 

where f n = ^- J* f(x)e~J nx dx. Define 

A' 



VnW = J^ /jW. 



1 

7 

It can be shown that 



1 /■* 



on(,x)=—\ f(x-t)K N (t)dt. 
2jt ./_„ 

It is also possible to prove that on(x) -> /(x) uniformly on [— jt, 7r] if /(*) 
is continuous. This is often called Fejer's theorem.] 

3.12. Repeat the analysis of Example 3.12 for /l(0). 

3.13. If f{t) e L 2 (0, In) and f(t) = £ neZ /„^" ( , show that 

*2tz 
lit Jq 



j r2jT °° 

—J \f(t)\ 2 dt= J2 i/»i 
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This relates the energy/power of f(t) to the energy/power of its Fourier 
series coefficients. 

[Comment: For example, if f(t) is one period of a 2jt -periodic signal, then 
the power interpretation applies. In particular, suppose that f(t) — i(t) a 
current waveform, and that this is the current into a resistor of resistance R; 
then the average power delivered to R is 

2jr 



1 f 
Pav = — Ri 2 (t)dt.] 

2n Jo 



3.14. Use the result of Example 1.20 to prove that 

(-1)" 7T 



£ 



3.15. Prove that 



In + 1 4 

n=0 



1 f sin[(L + i)r] 1 

dt — -. 



f 

Jo 



2tt Jo sinf^f] 

3.16. Use mathematical induction to prove that 

(1.1)" > 1 + j^jn 

for all n e N. 

3.17. Use mathematical induction to prove that 

(1 + hf >l+nh 

for all n e N, with h > — 1, (This is Bernoulli's inequality.) 

3.18. Use mathematical induction to prove that 4" + 2 is a multiple of 6 for all 

n € N. 

3.19. Conjecture a formula for (« € N) 

n 

s„ = y— — , 
ti k(k+l) 

and prove it using mathematical induction. 

3.20. Suppose that f(x) — tanx. Use (3.55) to approximate f(x) for all x e 
(—1, 1). Find an upper bound on |e(x)|, where e(x) is the error of the 
approximation. 
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3.21. Given f(x) = x 2 + 1, find all £ € (1, 2) such that 

/ (1) (?) 



r(1) /(2)-/(l) 



2- 1 

3.22. Use (3.65) to find an approximation to Vl + x for ie(|, |). Find an upper 
bound on the magnitude of e(x). 

3.23. Using a pocket calculator, compute (0.97) '/ 3 for n — 3 using (3.82). Find 
upper and lower bounds on the error e n +\(x). 

3.24. Using a pocket calculator, compute [1.05] 1 / 4 using (3.82). Choose n — 2 (i.e., 
quadratic approximation). Estimate the error involved in doing this using the 
error bound expression. Compare the bounds to the actual error. 

3.25. Show that 

= 2". 






3.26. Show that 








-I 


« - 


- 1 )- 


_(» 


j\ 


v J ~ 


■lj" 


"W 


for j = 1, 2, ...,« — 1, n. 








3.27. Show that for n > 2 












3.28. For 



= «2"" 
\ K I 



P„(k)- ( I )p k (l- p )"- k ,0<p< 1, 



where k = 0, 1, . . . , n, show that 

(n — k)p 

p n (k+ 1) = -/?„(£)■ 

(fc+ 1)(1 - p) 

[Comment: This recursive approach for finding p n (k) extends the range of 
n for which p n {k) may be computed before experiencing problems with 
numerical errors.] 

3.29. Identity (3.103) was confirmed using an argument associated with Fig. 3.7. A 
somewhat different approach is the following. Confirm (3.103) by working 
with 

1 f°° i I 2 1 f°° f°° i i 
-L / e- xl ' 2 dx =— / e-^ + y 2) ' 2 dxdy. 

I Z7T J —oo J ^- J— OQ t/— 00 
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(Hint: Use the Cartesian to polar coordinate conversion x — rcos6,y — 
r sin0.) 

3.30. Find the Maclaurin (infinite) series expansion for f(x) — sin -1 x. What is 
the radius of convergence? [Hint: Theorem 3.7 and Eq. (3.82) are use- 
ful.] 

3.31. From Eq. (3.85) for x > -1 

\og e (l+x) = J2^^- k + r(x), (3.P.1) 

k = \ 



where 

\r(x)\ 



-TTX n+ \ x>0 

n + l ' — 



-1 <x <0 



(3.P.2) 



(1-|jc|)(b+1)' 

[Eq. (3.86)]. For what range of values of x does the series 

(-l) k - l x k 



log e (l+x) = ^ 



k=\ 



converge? Explain using (3. P. 2). 
3.32. The following problems are easily worked with a pocket calculator, 
(a) It can be shown that 



(-lr 

J W = E (2k+lV x2k+1 + e2n+3(x) ' (3 ' P3) 



where 

(2n + 3)l 



e 2n+ 3(x)\ < - l ^A x\ 2n+ \ (3.P.4) 



Use (3. P. 4) to compute sin(x) to 3 decimal places of accuracy for x — 1.5 
radians. How large does n need to be? 
(b) Use the approximation 



(_i)"-V 

\og e (\+x)^Y.-^- — 



n = \ 



to compute log e (1.5) to three decimal places of accuracy. How large 
should N be to achieve this level of accuracy? 
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3.33. Assuming that 

2 f" sinx ^— v 

— / dx — 2 / 

Wo * ^ 

show that for suitable n 



jt lr (-i) r 



(2r + l)(2r + l)! 



2n 



2n + l 



(2n + 2)(2n + 2)\ 



(3.P.5) 



2 r* 

x Jo 



sinx 



■ dx > 1.17. 



What is the smallest n needed? Justify Eq. (3.P.5). 
3.34. Using integration by parts, find the asymptotic expansion of 



J X 



c(x) = I cost dt. 
3.35. Using integration by parts, find the asymptotic expansion of 



s(x) 



-f 

J x 



sin? dt. 



3.36. Use MATLAB to plot (on the same graph) function K^{x) in Problem 3.11 

for N = 2, 4, 15. 
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4 Linear Systems of Equations 



4.1 INTRODUCTION 

The necessity to solve linear systems of equations is commonplace in numerical 
computing. This chapter considers a few examples of how such problems arise 
(more examples will be seen in subsequent chapters) and the numerical problems 
that are frequently associated with attempts to solve them. We are particularly inter- 
ested in the phenomenon of ill conditioning. We will largely concentrate on how 
the problem arises, what its effects are, and how to test for this problem. 1 In addi- 
tion to this, we will also consider methods of solving linear systems other than the 
Gaussian elimination method that you most likely learned in an elementary linear 
algebra course. 2 More specifically, we consider LU and QR matrix factorization 
methods, and iterative methods of linear system solution. The concept of a singular 
value decomposition (SVD) is also introduced. 

We will often employ the term "linear systems" instead of the longer phrase 
"linear systems of equations." However, the reader must be warned that the phrase 
"linear systems" can have a different meaning from our present usage. In signals 
and systems courses you will most likely see that a "linear system" is either a 
continuous-time (i.e., analog) or discrete-time (i.e., digital) dynamic system whose 
input/output (I/O) behavior satisfies superposition. However, such dynamic systems 
can be described in terms of linear systems of equations. 



4.2 LEAST-SQUARES APPROXIMATION AND LINEAR SYSTEMS 

Suppose that f(x), g(x) e L 2 [0, 1], and that these functions are real-valued. Recall- 
ing Chapter 1, their inner product is therefore given by 

(/,*}= f f(x)g(x)dx. (4.1) 

Jo 

Methods employed to avoid ill-conditioned linear systems of equations will be mainly considered in 
a later chapter. These chiefly involve working with orthogonal basis sets. 

Review the Gaussian elimination procedure now if necessary. It is mandatory that you recall the basic 
matrix and vector operations and properties [(AB) = B A , etc.] from elementary linear algebra, 
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Note that here we will assume that all members of L 2 [0, 1] are real-valued (for 
simplicity). Now assume that {<p n (x)\n e Z^} form a linearly independent set such 
that 4> n (x) e L 2 [0, 1] for all n. We wish to find the coefficients a n such that 

AT— 1 

f(x) « J2 a n^n(x) for x e [0, 1]. (4.2) 

n=0 

A popular approach to finding a n [1] is to choose them to minimize the functional 

N-l 

V(a) = \\f(x)-J2 a n<Pn(x)\\ 2 , (4.3) 

n=0 

where, for future convenience, we will treat a n as belonging to the vector a — 
[a ai ■■■a N _ 2 a N -i\ T e R N . Of course, in (4.3) ||/|| 2 = (/, /) via (4.1). We are 
at liberty to think of 

N-l 

e(x) = f(x) - J2 an<t>nix) (4.4) 

n=0 

as the error between fix) and its approximation ^ n a n $ n ix). So, our goal is to 
pick a to minimize ||e(x)|| 2 , which, we have previously seen, may be interpreted 
as the energy of the error e(x). This methodology of approximation is called least- 
squares approximation. The version of this that we are now considering is only 
one of a great many variations. We will see others later. 
We may rewrite (4.3) as follows: 

N-l 

|2 



V(a) = ||/(x)- J>„0„(x)|| 2 

M = 

(N-l N-l \ 

f -^2 fl «0>" f-^2 ak $ k ) 
n=0 k=0 I 

= </,/>-(/,2>*&) 

\ k=0 I 

JN-1 \ IN -I N-l \ 

~ ( ^ a n4>n, /) + ( 5Z fl "^"' ^2 ak< t >k 
\«=0 / \n=0 k=0 I 

N-l N-l 

= \\f\\ 2 -J2a k {f,<t> k )-Y i a n {(t> n ,f) 

k=0 n=0 

N-l N-l 

+ ^2^2 a k a n ((j> k ,<l> n ) 



n=0 k=0 
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N-l 



= II/H 2 - 2 £>*(/,</>*} 

N-l N-l 
+ XI 5Z a k®n{<Pk,<Pn)- 



(4.5) 



n=0 k=0 



Naturally, we have made much use of the inner product properties of Chapter 1 . 
It is very useful to define 



P = 11/11, gk = {f,<Pk), r n ,k = {fa,fa), 
along with the vector 



g = [gogi ■■■gN-if e R^, 



and the matrix 



R = [r n ,k] € R 



NxN 



(4.6) 



(4.7a) 



(4.7b) 



We immediately observe that R — R T (i.e., R is symmetric). This is by virtue of 
the fact that r n ^ — (<p n , fa) = (fa, fa.) — r% n . Immediately we may rewrite (4.5) 
in order to obtain the quadratic form" 



V(a) = a T Ra — 2a T g + p. 



(4.8) 



The quadratic form occurs very widely in optimization and approximation prob- 
lems, and so warrants considerable study. An expanded view of R is 



R = 



/' 

Jo 



(pnix) dx 



/„ 



/' 

Jo 



</>o(x)(/>i(x) dx 
1 



(po{x)cj)\(x) dx 



Jo 



</>i (x) dx 



i: 
i: 



<t>o(x)<t>N-i(x)dx 



<Pl(x)<t>N-l(x)dx 



/ 4>o(x)4>N-l(x)dx / (pi(x)<p N _i(x)dx ■■■ \ (j>j,_,(x)dx 

■Jo Jo Jo 

(4.9) 

Note that the reader must get used to visualizing matrices and vectors in the general 

manner now being employed. The practical use of linear/matrix algebra demands 

this. Writing programs for anything involving matrix methods (which encompasses 

a great deal) is almost impossible without this ability. Moreover, modern software 

tools (e.g., MATLAB) assume that the user is skilled in this manner. 

The quadratic form is a generalization of the familiar quadratic ax + bx + c, for which x e C (if 
we are interested in the roots of a quadratic equation; otherwise we usually consider x s R) , and also 
a, b, c e R. 
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It is essential to us that i? _1 exist (i.e., we need R to be nonsingular). Fortu- 
nately, this is always the case because {(f> n \n e Z^} is an independent set. We shall 
prove this. If R is singular, then there is a set of coefficients a, such that 

JV-l 

£a/(&,^} = (4.10) 

for j e Zjsi . This is equivalent to saying that the columns of R are linearly depen- 
dent. Now consider the function 

N-l 



f(x) = ^2,cti<t>i{x) 



1=0 

so if (4.10) holds for all j e Z N , then 

IN-l \ N-l 



(/■ 0/> = H a i<$>u4>j ) = S «i (&. 0/> = 



\ i=0 / i'=0 

for all j € Zjv. Thus 



iV-l JV-l 



'N-l 



Y^OCi((j)i,(t)j) 



= 0, 



.7=0 ;=0 L ,'=0 

implying that 

(N-l N-l \ 

^2 <Xi<j>i, ^aj<t>j) = 0, 
i=0 ;=0 / 

or in other words, (/, /) = ||/|| 2 = 0, and so f(x) — for all x e [0, 1]. This 
contradicts the assumption that {<j) n \n e Zyv} is a linearly independent set. So i? _1 
must exist. 

From (4.3) it is clear that (via basic norm properties) V(a) > for all a e R N . 
If we now assume that f(x) — (all x e [0, 1]), we have p — 0, and g — too. 
Thus, a T Ra > for all a. 

Definition 4.1: Positive Semidefinite Matrix, Positive Definite Matrix Sup- 
pose that A — A T and that A e R" x ". Suppose that x e R". We say that A is 
positive semidefinite (psd) iff 

x 1 Ax > 
for all x. We say that A is positive definite (pd) iff 

x T Ax > 
for all x # 0. 
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If A is psd, we often symbolize this by writing A > 0, and if A is pd, we often 
symbolize this by writing A > 0. If a matrix is pd then it is clearly psd, but the 
converse is not necessarily true. 

So far it is clear that R > 0. But in fact R > 0. This follows from the linear 
independence of the columns of R. If the columns of R were linearly dependent, 
then there would be an a ^ such that Ra — 0, but we have already shown that 
7? _1 exists, so it must be so that Ra — iff a — 0. Immediately we conclude that 
R is positive definite. 

Why is R > so important? Recall that we may solve ax 2 + bx + c — (x € C) 
by completing the square: 



ax + bx 



b 

2d 



12 b 2 

+ c-— . (4.11) 

4a 



Now if a > 0, then y(x) — ax 2 + bx + c has a unique minimum. Since y^'(x) — 
lax + b — Qior x — — j-, this choice of x forces the first term of (4.1 1) (right-hand 
side of the equality) to zero, and we see that the minimum value of y(x) is 

/ b \ _ b 2 _ Aac - b 2 

y \2a\)~ C ~A~a'~ ~4~a ' 

Thus, completing the square makes the location of the minimum (if it exists), and 
the value of the minimum of a quadratic very obvious. For the same purpose we 
may complete the square of (4.8): 

V(a) = [a - R- l g] T R[a - R~ l g] + p- g T R~ l g. (4.13) 

It is quite easy to confirm that (4.13) gives (4.8) (so these equations must be 
equivalent): 



-R~ l g] 


r R[a- 


R~ 1 g] + 


p-g T R~ 


l g 








= [a T - 


g T (R~ 


l ) T ][Ra- 


-gl + p- 


g T R~ 


l 8 






— a Ra 


-g T R 


Ra — a 


T g + g T R 


" 1 * + 


P~ 


8 T 


R 


= a Ra 


T 

-g a 


-a T g + 


p — a Ra 


-2a 7 


g + 


p. 





We have used (R ! ) r = (R T ) l = R \ and the fact that a T g — g T a. If vector 
x — a — R~ l g, then 

[a - R~ l g] T R[a - i? _1 g] = x T Rx. 

So, because R > 0, it follows that x T Rx > for all x ^ 0. The last two terms of 
(4.13) do not depend on a. So we can minimize V(a) only by minimizing the first 
term. R > implies that this minimum must be for x — 0, implying a — a, where 

a- R~ 1 g = 0, 

or 

Ra = g. (4.14) 
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Thus 

a — argminV(a). (4.15) 

We see that to minimize ||e(x)|| 2 we must solve a linear system of equations, 
namely, Eq. (4.14). We remark that for R > 0, the minimum of V(a) is at a unique 
location a e R N ; that is, the minimum is unique. 

In principle, solving least-squares approximation problems seems quite simple 
because we have systematic (and numerically reliable) methods to solve (4.14) 
(e.g., Gaussian elimination with partial pivoting). However, one apparent difficulty 
is the need to determine various integrals: 



= / f(x)(j>k(x)dx,r nyk = / (f>„( 
Jo Jo 



Usually, the independent set {4>k\k e Z^} is chosen to make finding r n k rela- 
tively straightforward. In fact, sometimes nice closed-form expressions exist. But 
numerical integration is generally needed to find gk- Practically, this could involve 
applying series expansions such as considered in Chapter 3, or perhaps using 
quadratures such as will be considered in a later chapter. Other than this, there 
is a more serious problem. This is the problem that R might be ill-conditioned. 



4.3 LEAST-SQUARES APPROXIMATION AND ILL-CONDITIONED 
LINEAR SYSTEMS 

A popular choice for an independent set {<j>k(x)} would be 

c/, k (x) = x k for x e [0, 1], k e Z N . (4.16) 

Certainly, these functions belong to the inner product space L 2 [0, 1]. Thus, for 
f(x) € L 2 [0, 1] an approximation to it is 

N-l 

f(x) = J2 «*** € P" -1 [0, 1], (4.17) 

fc=0 

and so we wish to fit a degree N — 1 polynomial to f(x). Consequently 

g k = [ x k f(x)dx (4.18a) 



which is sometimes called the kth moment^ of f(x) on [0, 1], and also 

•1 rl 

(j>„(x)<pk(x) dx = / 
'o Jo 

for n, k e Z#. 



r n ,k= / (f>n(x)<Pk(x)dx= x n+k dx= (4.18b) 

Jo Jo n + k + 1 



The concept of a moment is also central to probability theory. 
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For example, suppose that N — 3 (i.e., a quadratic fit); then 



-I T 



J f(x)dx I xf(x)dx J x f(x)dx 
Jo Jo Jo 



=gl 



= X2 



(4.19a) 



and 



R 



1 


1 

2 


1 

3 


1 

2 


1 
3 


1 

4 


1 

3 


1 

4 


1 

5 



roo roi ro2 
no r n r n 
no n\ m 



(4.19b) 



and a = [do «i 02] > so we wish to solve 



fl 



1 


1 
2 


1 
3 


1 


1 


1 


2 


3 


4 


1 


1 


1 


3 


4 


5 



/ f(x)dx 
Jo 

I xf(x)dx 
Jo 

If 

L Jo 



x f{x) dx 



(4.20) 



We remark that R does not depend on the "data" f(x), only the elements of g 
do. This is true in general, and it can be used to advantage. Specifically, if f(x) 
changes frequently (i.e., we must work with different data), but the independent 
set does not change, then we need to invert R only once. 

Matrix R in (4.19b) is a special case of the famous Hilbert matrix [2-4]. The 
general form of this matrix is (for any N € N) 

1 

N 

1 



R = 



1 








2 


3 


1 


1 


1 


2 


3 


4 


1 


1 


1 


3 


4 


5 



N+ 1 
1 

N + 2 



1 



1 



1 



1 



N N + 1 N + 2 



2N - 1 



€ R 



NxN 



(4.21) 



Thus, (4.20) is a special case of a Hilbert linear system of equations. The matrix R 
in (4.21) seems "harmless," but it is actually a menace from a numerical computing 
standpoint. We now demonstrate this concept. 
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Suppose that our data are something very simple. Say that 

f{x) = 1 for all x e [0, 1]. 

In this case gi = f x k dx — rry. Therefore, for any TV € N, we are compelled to 
solve 



1 


2 


1 


1 


2 


3 


1 


1 


3 


4 



1 



1 



1 

TV 
1 



TV + 1 
1 



N 



1 



L TV TV + 1 TV + 2 



2TV- 1 









r i "1 




fl 

a\ 

«2 





i 

2 
1 

3 




a-N-i 




1 

- TV - 



(4.22) 



A moment of thought reveals that solving (4.22) is trivial because g is the first 
column of R. Immediately, we see that 



a = [100- -OOY 



(4.23) 



(No other solution is possible since R~ exists, implying that Ra — g always 
possesses a unique solution.) 

MATLAB implements Gaussian elimination (with partial pivoting) using the 
operator "\" to solve linear systems. For example, if we want x in Ax — y for 
which A~ l exists, then x — A\y. MATLAB also computes R using function "hilb"; 
that is, R — hilb (TV) will result in R being set to a TV x TV Hilbert matrix. Using the 
MATLAB "\" operator to solve for a in (4.22) gives the expected answer (4.23) for 
TV < 50 (at least). The computer-generated answers are correct to several decimal 
places. (Note that it is somewhat unusual to want to fit polynomials to data that 
are of such large degree.) So far, so good. 

Now consider the results in Appendix 4. A. The MATLAB function "inv" may 
be used to compute the inverse of matrices. The appendix shows R _1 (computed 
via inv) for TV = 10, 11, 12, and the MATLAB computed product RR~ l for these 
cases. Of course, RR~ l — I (identity matrix) is expected in all cases. For the 
number of decimal places shown, we observe that RR~ l ^ /. Not only that, but 
the error E — RR~ X — I rapidly becomes large with an increase in TV. For TV = 12, 
the error is substantial. In fact, the MATLAB function inv has built-in features to 
warn of trouble, and it does so for case TV = 12. Since RR~ l is not being computed 
correctly, something has clearly gone wrong, and this has happened for rather small 
values of TV. This is in striking contrast with the previous problem, where we wanted 
to compute a in (4.22). In this case, apparently, nothing went wrong. 

We may consider changing our data to f(x) — x N ~ l . In this case gk — 1/(TV + 
k) for k e Z#. The vector g in this case will be the last column of R. Thus, 
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mathematically, a — [00 • • • 001] r . If we use MATLAB "\" to compute a for this 
problem we obtain the computed solutions: 

a = [0.0000 0.0000 0.0000 0.0000 . . . 

0.0002 -0.0006 0.0013 -0.00170.0014 -0.0007 l.OOOlf (N = 11), 

a = [0.0000 0.0000 0.0000 -0.0002 . . . 0.0015 -0.0067 

0.0187 -0.0342 0.0403 -0.0297 0.0124 0.9978] r (N = 12). 

The errors in the computed solutions a here are much greater than those experienced 
in computing a in (4.23). 

It turns out that the Hilbert matrix R is a classical example of an ill-conditioned 
matrix (with respect to the problem of solving linear systems of equations). The 
linear system in which it resides [i.e., Eq. (4.14)] is therefore an ill-conditioned 
linear system. In such systems the final answer (which is a here) can be exquisitely 
sensitive to very small perturbations (i.e., disturbances) in the inputs. The inputs 
in this case are the elements of R and g. From Chapter 2 we remember that R 
and g will not have an exact representation on the computer because of quantiza- 
tion errors. Additionally, as the computation proceeds rounding errors will cause 
further disturbances. The result is that in the end the final computed solution can 
deviate enormously from the correct mathematical solution. On the other hand, we 
have also shown that it is possible for the computed solution to be very close to 
the mathematically correct solution even in the presence of ill conditioning. Our 
problem then is to be able to detect when ill conditioning arises, and hence might 
pose a problem. 



4.4 CONDITION NUMBERS 

In the previous section there appears to be a problem involved in accurately com- 
puting the inverse of R (Hilbert matrix). This was attributed to the so-called ill 
conditioning of R. We begin here with some simpler lower-order examples that 
illustrate how the solution to a linear system Ax — y can depend sensitively on A 
and y. This will lead us to develop a theory of condition numbers that warn us that 
the solution x might be inaccurately computed due to this sensitivity. 

We will consider Ax = y on the assumption that A e R" x ", and x, y e R". 
Initially we will assume n — 2, so 



fl00 floi 
flio an 









(4.24) 



In practice, we may be uncertain about the accuracy of the entries of A and y. 
Perhaps these entities originate from experimental data. So the entries may be 
subject to experimental errors. Additionally, as previously mentioned, the elements 
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of A and y cannot normally be exactly represented on a computer because of the 
need to quantize their entries. Thus, we must consider the perturbed system 



flOO G01 

flio an 



<Saoo <5floi 
&ci\q San 



=/>A 






>'o 



Syo 

Sy\ 

=Sy 



(4.25) 



The perturbations are SA and Sy. We will assume that these are "small." As you 
might expect, the practical definition of "small" will force us to define and work 
with suitable norms. This is dealt with below. We further assume that the computing 
machine we use to solve (4.25) is a "magical machine" that computes without 
rounding errors. Thus, any errors in the computed solution, denoted x here, can be 
due only to the perturbations SA and Sy. It is our hope that x « x. Unfortunately, 
this will not always be so, even for n — 2 with small perturbations. 

Because n is small, we may obtain closed-form expressions for A -1 , x, [A + 
S A] -1 , and x. More specifically 



floofln — floiflio 



an 

-flio 



-fl l 

floo 



(4.26) 



and 



[A + SA]~ [ = 



1 



(floo + <5fl o)(flii + San) - (floi + S«oi)(«io + Saw) 



an + <5fln 
-(flio + 5flio) 



-(floi +5fl i) 
(aoo + <5floo) 



(4.27) 



The reader can confirm these by multiplying A as given in (4.26) by A. The 
2x2 identity matrix should be obtained. Using these formulas, we may consider 
the following example. 

Example 4.1 Suppose that 



-.01 
.01 



Nominally, the correct solution is 



1 
100 



Let us consider different perturbation cases: 
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1. Suppose that 



In this case 



2. Suppose that 



for which 



SA = 




.005 



Sy = [0 or 



jc = [1.1429 -85.7143]' 



<5A = 



.03 



SA 



sy = to or 



.01 

.02 



This matrix is mathematically singular, so it does not possess an inverse. If 
MATLAB tries to compute x using (4.27), then we obtain 

x = 1.0 x 10 17 x [-0.0865 -8.6469] r 

and MATLAB issues a warning that the answer may not be correct. Obvi- 
ously, this is truly a nonsense answer. 
3. Suppose that 



SA 





-.02 



.Sy = [0.10 -0.05]'. 



In this case 



ic = [-1.1500 -325.0000] 7 ". 



It is clear that small perturbations of A and y can lead to large errors in the 
computed value for x. These errors are not a result of accumulated rounding errors 
in the computational algorithm for solving the problem. For computations on a 
"nonmagical" (i.e., "real") computer, this should be at least intuitively plausible 
since our formulas for x are very simple in the sense of creating little opportunity 
for rounding error to grow (there are very few arithmetical operations involved). 
Thus, the errors x — x must be due entirely (or nearly so) to uncertainties in the 
original inputs. We conclude that the real problem is that the linear system we 
are solving is too sensitive to perturbations in the inputs. This naturally raises the 
question of how we may detect such sensitivity. 

In view of this, we shall say that a matrix A is ill-conditioned if the solution x 
(in Ax = y) is very sensitive to perturbations on A and y. Otherwise, the matrix 
is said to be well-conditioned. 
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We will need to introduce appropriate norms in order to objectively measure 
the sizes of objects in our problem. However, before doing this we make a few 
observations that give additional insight into the nature of the problem. In Example 
4.1 we note that the first column of A is big (in some sense), while the second 
column is small. The smallness of the second column makes A close to being 
singular. A similar observation may be made about Hilbert matrices. For a general 
TV x TV Hilbert matrix, the last two columns are given by 



1 



TV- 1 
1 

TV 



1 



1 

TV 
1 



TV + 1 



1 



L 2TV - 2 2N - 1 J 



For very large TV, it is apparent that these two columns are almost linearly depen- 
dent; that is, one may be taken as close to being equal to the other. A simple 
numerical example is that j4, « -j^y. Thus, at least at the outset, it seems that 
ill-conditioned matrices are close to being singular, and that this is the root cause 
of the sensitivity problem. 

We now need to extend our treatment of the concept of norms from what we 
have seen in earlier chapters. Our main source is Golub and Van Loan [5], but 
similar information is to be found in Ref. 3 or 4 (or the references cited therein). 
A fairly rigorous treatment of matrix and vector norms can be found in Horn and 
Johnson [6]. 

Suppose again that x e R". The p-norm of x is defined to be 



11*11,= 



5>*l' 



.k=0 



Up 



(4.28) 



where p > 1. The most important special cases are, respectively, the 1-norm, 2- 
norm, and oo-norm: 





71-1 


l*lll 


= £l**l. 




k=Q 


l*l| 2 


= U T x]"\ 


*||oo 


= max \x k \. 

0<k<n-\ 



(4.29a) 

(4.29b) 
(4.29c) 



The operation "max" means to select the biggest \xk\- A unit vector with respect 
to norm II • II is a vector x such that llxll = 1. Note that if x is a unit vector with 
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respect to one norm, then it is not necessarily a unit vector with respect to another 
choice of norm. For example, suppose that x = [-^j] r ; then 

V3+ 1 V3 

I |x|| 2 = i,IM|i = ^— ,IMIoo = — • 

The vector x is a unit vector under the 2-norm, but is not a unit vector under the 
1-norm or the oo-norm. 

Norms have various properties that we will list without proof. Assume that 
x,y e R". For example, the Holder inequality [recall (1.16) (in Chapter 1) for 
comparison] is 

\x T y\ < \\x\\ p \\y\\ q (4.30) 

for which \- + -7 = 1. A special case is the Cauchy-Schwarz inequality 

\x T y\<\\x\h\\y\\2, (4.31) 

which is a special instance of Theorem 1.1 (in Chapter 1). An important feature of 
norms is that they are equivalent. This means that if || • ||„ and || • ||« are norms 
on R", then there are c\, C2 > such that 

ci\\x\\ a < 11*11/1 <C2||jc||„ (4.32) 

for all x € R" . Some special instances of this are 

ll*||2<ll*||i<-A||*||2, ( 433a ) 

Hx||oo<IWl2<-AlWloo, (4.33b) 

IMIco <Nli <n|Mlcc (4.33c) 

Equivalence is significant with respect to our problem in the following manner. 
When we define condition numbers below, we shall see that the specific value of 
the condition number depends in part on the choice of norm. However, equivalence 
says that if a matrix is ill-conditioned with respect to one type of norm, then it 
must be ill-conditioned with respect to any other type of norm. This can simplify 
analysis in practice because it allows us to compute the condition number using 
whatever norms are the easiest to work with. Equivalence can be useful in another 
respect. If we have a sequence of vectors in the space R" , then, if the sequence is 
Cauchy with respect to some chosen norm, it must be Cauchy with respect to any 
other choice of norm. This can simplify convergence analysis, again because we 
may pick the norm that is easiest to work with. 

In Chapter 2 we considered absolute and relative error in the execution of 
floating-point operations. In this setting, operations were on scalars, and scalar 
solutions were generated. Now we must redefine absolute and relative error for 
vector quantities using the norms defined in the previous paragraph. Since x € R" 
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is the computed (i.e., approximate) solution to x e R" it is reasonable to define the 
absolute error to be 

€ a = \\x-x\\, (4.34a) 



and the relative error is 



||x-x|| 
er = li - n - n - Li . (4.34b) 

11*11 



Of course, x ^ is assumed here. The choice of norm is in principle arbitrary. 
However, if we use the oo-norm, then the concept of relative error with respect to 
it can be made equivalent to a statement about the correct number of significant 
digits in x: 

Hi ~ X|100 « l(T d . (4.35) 

\\x\\oo 

In other words, the largest element of the computed solution x is correct to approx- 
imately d decimal digits. For example, suppose that x = [1.256 — 2.554] r , and 
x = [1.251 - 2.887] r ; then x-x = [-0.005 - 0.333] r , and so 

||x-x||oo = 0.333, 11x11^ = 2.554, 

so therefore e r — 0.1304 « 10 _1 . Thus, x has a largest element that is accurate to 
about one decimal digit, but the smallest element is observed to be correct to about 
three significant digits. 

Matrices can have norms defined on them. We have remarked that ill condi- 
tioning seems to arise when a matrix is close to singular. Suitable matrix norms 
can allow us to measure how close a matrix is to being singular, and thus gives 
insight into its condition. Suppose that A, B e R mx " (so A and B are not neces- 
sarily square matrices). || • |||R mx " -> R is a matrix norm, provided the following 
axioms hold: 

(MN1) ||A|| > for all A, and ||A|| = iff A = 0. 
(MN2) ||A + fl||<||A|| + ||B||. 

(MN3) 1 1 a A|| = | a | ||A||. Constanta is from the same field as the elements of 
the matrix A. 

In the present context we usually consider a e R. Extensions to complex-valued 
matrices and vectors are possible. The axioms above are essentially the same as for 
the norm in all other cases (see Definition 1.3 for comparison). The most common 
matrix norms are the Frobenius norm 



\\A\\ F 



m—\ n—\ 



EIj«ul 2 (4.36a) 



\ k=0 1=0 
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and the p-norms 

\\AX\\ P 

||A|| p = sup^^. (4.36b) 

x=iO \\X\\p 

We see that in (4.36b) the matrix p-norm is dependent on the vector /?-norm. Via 
(4.36b), we have 

\\Ax\\ p < \\A\\ p \\x\\ p . (4.36c) 

We may regard A as an operator applied to x that yields output Ax. Equation 
(4.36c) gives an upper bound on the size of the output, as we know the size of A 
and the size of x as given by their respective p-norms. Also, since A e R mx " it 
must be the case that x e R" , but y € R™ . We observe that 



\A\\ P = sup 
xjtO 



11*11. 



= max \\Ax\\ p . (4.37) 

p \\x\\ p =\ 



This is an alternative means to compute the matrix p-norm: Evaluate ||Ajt||n at all 
points on the unit sphere, which is the set of vectors {x|||x||p = 1}, and then pick 
the largest value of ||A;e|L. Note that the term "sphere" is an extension of what 
we normally mean by a sphere. For the 2-norm in n dimensions, the unit sphere is 
clearly 

\\x\\2=[xl+xl + --- + x 2 n _ l \ l l 2 = \. (4.38) 

This represents our intuitive (i.e., Euclidean) notion of a sphere. But, say, for the 
1-norm the unit sphere is 

Iklli = |*oH-l*il + -" + l*».-il = i. (4-39) 

Equations (4.38) and (4.39) specify very different looking surfaces in n-dimensional 
space. A suggested exercise is to sketch these spheres for n — 2. 

As with vector norms, matrix norms have various properties. One property pos- 
sessed by the matrix p-norms is called the submultiplicative property: 

\\AB\\ P < \\A\\ P \\B\\ P AeR mxn , B e R" x? . (4.40) 

(The reader is warned that not all matrix norms possess this property; a coun- 
terexample appears on p. 57 of Golub and Van Loan [5]). A miscellany of other 
properties (including equivalences) is 

l|A|| 2 < ||A|| F < yfr\\A\\ 2 , (4.41a) 

max \a,j\ < \\A\\2 < V OTMmax \ a ij I, (4.41b) 
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m — \ 



A||i=maxy]|a u |, (4.41c) 



/eZ„ 

(=0 



n-\ 

X 

ieZ, 

1 



A||oo =maxY"|a iJ |, (4.41d) 

Sal '—' J 

.7=0 



,-JAHoo < ||A|j 2 < V^IIAMoo, (4.41e) 

l|A||i <||A|| 2 <V^||A||i. (4.41f) 

m 



The equivalences [e.g., (4.41a) and (4.41b)] have the same significance for 
matrices as the analogous equivalences for vectors seen in (4.32) and (4.33). 

From (4.41c,d) we see that computing matrix 1 -norms and oo-norms is easy. 
However, computing matrix 2-norms is not easy. Consider (4.37) with p — 2: 

||A|| 2 = max ||Ax|| 2 . (4.42) 

11*112 = 1 

Let R — A T A e R" x " (no, R is not a Hilbert matrix here; we have "recycled" the 
symbol for another use), so then 

\\Ax\\\ = x T A T Ax = x T Rx. (4.43) 

Now consider n — 2. Thus 

(4.44) 



x Rx — [xqxi] 



roo r 0i 

no n\ 



Xq 



where rgi = no because R — R T . The vectors and matrix in (4.44) multiply out 
to become 

x T Rx — roo*o + 2r Q \XQXi +r n x\. (4.45) 

Since \\A\\\ — max|| x || 2 — i ||Ajt|||, we may find \\A\\\ by maximizing (4.45) subject 
to the equality constraint 1 1 jc 1 1^ — 1) i- e -> x 1 x = xi + x\ = \. This problem may 
be solved by using Lagrange multipliers (considered somewhat more formally in 
Section 8.5). Thus, we must maximize 

V(x) =x T Rx-X[x T x- 1], (4.46) 

where k is the Lagrange multiplier. Since 

V(x) = r m xl + 2r 0l x x 1 + r n x\ - X[xq + x\ - 1], 
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we have 



dV(x) 

dx 

dv(x) 

dx\ 



= 2rooxo + 2roix\ — 2Axo = 0, 
= 2roixo + 2r\\x\ — 2Xx\ — 0, 



and these equations may be rewritten in matrix form as 



HX) r 0i 

no n\ 



XQ 

x\ 



xo 
x\ 



(4.47) 



In other words, Rx — Xx, Thus, the optimum choice of x is an eigenvector of 
R — A T A. But which eigenvector is it? 

First note that A -1 exists (by assumption), so x T Rx — x T A T Ax — 
(Ax) T (Ax) > for all x ^ 0. Therefore, R > 0. Additionally, R = R T , so all 
of the eigenvalues of R are real numbers. 5 Furthermore, because R > 0, all of its 
eigenvalues are positive. This follows if we consider Rx — Xx, and assume that 
X < 0. In this case x T Rx — Xx T x — X\\x\\2 < for any x ^ 0. [If X — 0, then 
Rx — • x — implies that x = (as 7? _1 exists), so x T Rx — 0.] But this con- 
tradicts the assumption that R > 0, and so all of the eigenvalues of R must be 
positive. Now, since ||Ax||2 = x T A T Ax — x T Rx — x T (Xx) — A||x|||, and since 
Hxllj = 1, it must be the case that IIAxllj is biggest for the eigenvector of R cor- 
responding to the biggest eigenvalue of R. If the eigenvalues of R are denoted X\ 
and Xq with A.i > Xq > 0, then finally we must have 



||A||| = Ai. 



(4.48) 



This argument can be generalized for all n > 2. If R > 0, we assume that all 
of its eigenvalues are distinct (this is not always true). If we denote them by 
Xq, Xi, . . . , X n -i, then we may arrange them in decreasing order: 



A.,, 



> X„-2 > ■ ■ ■ > X\ > Xq > 0. 



(4.49) 



Therefore, for A € R" 



|A||2=A„_i. 



(4.50) 



The problem of computing the eigenvalues and eigenvectors of a matrix has its own 
special numerical difficulties. At this point we warn the reader that these problems 
must never be treated lightly. 



If A is a real-valued symmetric square matrix, then we may prove this claim as follows. Suppose 
that for eigenvector x of A, the eigenvalue is A, that is, Ax = Xx. Now ((Ax)*) = ((kx)*) , and so 
(x*) T A T = k*(x*) T . Therefore, (x*) T A T x = X*(x*) T x. But (x*) T A T x = (x*) T Ax = X(x*) T x, so 



finally X*(x*) T a 



; X(x*) x, so we must have X = X*. This can be true only if X e R. 
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Example 4.2 Let det(A) denote the determinant of A. Suppose that R — A T A, 

where 

1 0.5 

t 0.5 1 
We will find ||A|| 2 . Consider 



R = 



1 0.5 
0.5 1 



x 
x\ 



XQ 

X] 



We must solve det(A7 — R) — for A. [Recall that det(A7 — R) is the characteristic 
polynomial of R.] Thus 



det (XI - R) - det 

and (X - l) 2 - \ = X 2 - 2X 



A-l -0.5 
-0.5 X - 1 

\ = 0, for 



= (X - I) 2 



1 



= 0, 



A = 



(-2) ±J (-2) 2 -4-1 



2± 1 



So, A.1 



X Q 



2- 1 
Thus, 1 1 A|| 2 = A.i 



1 3 
2' 2' 



|A|| 2 



and so finally 



(We do not need the eigenvectors of R to compute the 2-norm of A.) 

We see that the essence of computing the 2-norm of matrix A is to find the zeros 
of the characteristic polynomial of A T A. The problem of finding polynomial zeros 
is the subject of a later chapter. Again, this problem has its own special numerical 
difficulties that must never be treated lightly. 

We now derive the condition number. Begin by assuming that A € R" x ", and 
that A -1 exists. The error between computed solution x to Ax — y and x is 

e = x-x. (4.51) 

Ax — y, but Ax ^ y in general. So we may define the residual 

r — y — Ax. 

We see that 

Ae — Ax — Ax — y — Ax — r. 

Thus, e — A~ l r. We observe that if e — 0, then r — 0, but if r is small, then e 
is not necessarily small because A -1 might be big, making A~V big. In other 



(4.52) 



(4.53) 
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words, a small residual r does not guarantee that x is close to x. Sometimes r is 
computed as a cursory check to see if x is "reasonable." The main advantage of 
r is that it may always be computed, whereas x is not known in advance and so 
e may never be computed exactly. Below it will be shown that considering r in 
combination with a condition number is a more reliable method of assessing how 
close x is to x. 

Now, since e — A~V, we can say that \\e\\ p = ||A _1 r|| /:1 < ||A _1 ||p||r||p [via 
(4.36c)]. Similarly, since r = Ae, we have \\r\\ p = \\Ae\\ p < \\A\\ p \\e\\ p . Thus 

lln ' 1 " <IMI„ ^IIA-'lUML. (4.54) 



\\A\\ P 
Similarly, x — A~ l y, so immediately 



N ' P < \\x\\ P <\\A- l \\ p \\y\\ p . (4.55) 



l|A||_ 
If ||x|L ^ 0, and \\y\\ p ^ 0, then taking reciprocals in (4.55) yields 

\\A 



< — — . (4.56) 



\\A- l \\ p \\y\\ p - \\x\\ p - \\y\\ p 
We may multiply corresponding terms in (4.56) and (4.54) to obtain 

i ikiip ik|L 114 _, ikn 



IIA-'llpllAHp \\y\\ p \\x\\ p \\y\\p 

We recall from (4.34b) that e r = ll *r* llp = ^K so 

11-^11/? I I -* I I p 

!,!„,„ ,'S', P Kr< IIA-'II.IIAH^. (4.58) 



We call 



\\A- l \\ p \\A\\ p \\y\\ p ~ ••"•• •••'\\y\\ 

\\r\\ P lly-Ax||, 



(4.59) 



\\y\\ P \\y\\p 

the relative residual. We define 

K p (A) = \\A\\p\\A- 1 \\ p (4.60) 

to be the condition number of A. It is immediately apparent that k p {A) > 1 for 
any A and valid p. We see that e r is between 1/k p (A) and K p (A) times the 
relative residual. In particular, if k p (A) >> 1 (i.e., if the condition number is very 
large), even if the relative residual is tiny, then e r might be large. On the other 
hand, if k p (A) is close to unity, then e r will be small if the relative residual is 
small. In conclusion, if K p (A) is large, it is a warning (not a certainty) that small 
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perturbations in A and y may cause x to differ greatly from x. Equivalently, ifK p (A) 
is large, then a small r does not imply that x is close to x. 

A rule of thumb in interpreting condition numbers is as follows [3, p. 229], and 
is more or less true regardless of p in (4.60). If K p {A) « d x 10*, where d is a 
decimal digit from one to nine, we can expect to lose (at worst) about k digits of 
accuracy. The reason that p does not matter too much is because we recall that 
matrix norms are equivalent. Therefore, for this rule of thumb to be useful, the 
working precision of the computing machine/software package must be known. 
For example, MATLAB computes to about 16 decimal digits of precision. Thus, 
k > 16 would give us concern that x is not close to x. 



Example 4.3 Suppose that 

A = 



1 1-e 

1 1 



€ R 2x2 , |e| << 1. 



We will determine an estimate of ati(A). Clearly 



6 



1 -1 + 6 
1 1 



*oo boi 
ho b\\ 



We have 

l 



l 



^ |a/,o| = laool + l«iol = 2, y^|a,-,i| = |«oi I + l«n| = |1 -e| + 1, 
i'=0 i=0 

so via (4.41c), \\A\\ X = max{2, |1 - e| + 1} « 2. Similarly 
i -. i 



T \bi,o\ = \boo\ + \ho\ = r7>f] \Ki\ = 1% I + l^ii I = A 



/=o 



kl 



1=0 



kl 



-1 + e 



so again via (4.41c) || A ||i=max|A, 



1-i 



h\ « A. Thus 



*i(A) 



v\ 



We observe that if e = 0, then A 
is a reasonable result because 



does not exist, so our approximation to k\ (A) 



lim k\(A) = oo. 



We may wish to compute kj{A) = ||A||2||A _ ||2. We will suppose that A e 
R" x " and that A -1 exists. But we recall that computing matrix 2-norms involves 
finding eigenvalues. More specifically, \\A\\i is the largest eigenvalue of R — A 7 A 
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[recall (4.50)]. Suppose, as in (4.49), that Xq is the smallest eigenvalue of R for 



which the corresponding eigenvector is denoted by v, that is, Rv — Xqv. Then we 

4' ' 



observe that R l v — j-V. In other words, 1/Aq is an eigenvalue of R '.By similar 



reasoning, l/Xk for k e Z„ must all be eigenvalues of 7? _1 . Thus, 1/Ao will be the 
biggest eigenvalue of R~ l . For present simplicity assume that A is a normal matrix. 
This means that AA T — A T A — R. The reader is cautioned that not all matrices A 
are normal. However, in this case we have R~ l — A~ l A~ T — A~ T A~ l . [Recall 
that (A _1 ) r = (A r ) _1 = A~ T .] We have that IIA -1 !^ is the largest eigenvalue of 
A~ T A~ l , but R _1 — A~ T A -1 since A is assumed normal. The largest eigenvalue 
of R~ l has been established to be 1/A.o, so it must be the case that for a normal 
matrix A (real-valued and invertible) 



*2(A) = t /-jp, (4.61) 

that is, A is ill-conditioned if the ratio of the biggest to smallest eigenvalue of 
A T A is large. In other words, a large eigenvalue spread is associated with matrix 
ill conditioning. It turns out that this conclusion holds even if A is not normal; that 
is, (4.61) is valid even if A is not normal. But we will not prove this. (The interested 
reader can see pp. 312 and 340 of Horn and Johnson [6] for more information.) 

An obvious difficulty with condition numbers is that their exact calculation often 
seems to require knowledge of A -1 . Clearly this is problematic since computing 
A -1 accurately may not be easy or possible (because of ill conditioning). We seem 
to have a "chicken and egg" problem. This problem is often dealt with by using 
condition number estimators. This in turn generally involves placing bounds on 
condition numbers. But the subject of condition number estimation is not within 
the scope of this book. The interested reader might consult Higham [7] for further 
information on this subject if desired. There is some information on this matter in 
the treatise by Golub and Van Loan [5, pp. 128-130], which includes a pseudocode 
algorithm for oo-norm condition number estimation of an upper triangular nonsin- 
gular matrix. We remark that ||A||2 is sometimes called the spectral norm of A, 
and is actually best computed using entities called singular values [5, p. 72]. This 
is because computing singular values avoids the necessity of computing A -1 , and 
can be done in a numerically reliable manner. Singular values will be discussed in 
more detail later. 

We conclude this section with a remark about the Hilbert matrix R of Section 4.3. 
As discussed by Hill [3, p. 232], we have 

K 2 (R) oc e aN 

for some a > 0. (Recall that symbol oc means "proportional to.") Proving this is 
tough, and we will not attempt it. Thus, the condition number of R grows very 
rapidly with N and explains why the attempt to invert R in Appendix 4. A failed 
for so small a value of N. 
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4.5 LU DECOMPOSITION 

In this section we will assume A € R" x ", and that A -1 exists. Many algorithms 
to solve Ax — y work by factoring the matrix A in various ways. In this section 
we consider a Gaussian elimination approach to writing A as 



A = LU, 



(4.62) 



where L is a nonsingular lower triangular matrix, and U is a nonsingular upper 
triangular matrix. This is the LU decomposition (factorization) of A. Naturally, 
L, U e R nx ", and L — [lij], U — [m,- j]. Since these matrices are lower and upper 
triangular, respectively, it must be the case that 

ll : — for j > i and «,- ,• = for j < i. (4.63) 

For example, the following are (respectively) lower and upper triangular matrices: 



1 
1 1 
1 1 1 



U 



1 2 3 
4 5 
6 



These matrices are clearly nonsingular since their determinants are 1 and 24, respec- 
tively. In fact, L is nonsingular iff /,-_,- ^ for all i , and U is nonsingular iff u ;> ; ^ 
for all j. We note that with A factored as in (4.62), the solution of Ax — y becomes 
quite easy, but the details of this will be considered later. We now concentrate on 
finding the factors L, U. 

We begin by defining a Gauss transformation matrix Gk such that 



G k x 



1 










"n-1 



• 


• " 




x 




• 

1 • 


• 

• 




Xk-1 
Xk 


= 


• 


• 1 _ 




Xfi—l 





xo 



Xk-l 





(4.64) 



for 



x} = 



Xj 
Xk-l 



,i = k, 



(4.65) 



The superscript k on x k . does not denote raising Xj to a power; it is simply part of 
the name of the symbol. This naming convention is needed to account for the fact 
that there is a different set of r values for every Gk ■ For this to work requires that 
Xk-i # 0. Equation (4.65) followed from considering the matrix-vector product 
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in (4.64): 



-r k Xk-i + x k — 
-r k+1 Xk-i + xt+i — 



l «-l 



Xk-l 



0. 



We observe that Gt is "designed" to annihilate the last n — k elements of vector 
x. We also see that G k is lower triangular, and if it exists, always possesses an 
inverse because the main diagonal elements are all equal to unity. A lower triangular 
matrix where all of the main diagonal elements are equal to unity is called unit 
lower triangular. Similar terminology applies to upper triangular matrices. Define 
the kth Gauss vector 



l «-lJ 



k zeros 



The £th unit vector is 



el =[0---01 0---0 ]. 



k zeros n—k—l zeros 



(4.66) 



(4.67) 



If / is an n x n identity matrix, then 



Gt k T 

k = I -x e k _ l 



(4.68) 



forfc= 1,2, 



1. For example, if n — 4, we have 



G, = 



Gi = 



1 



~rl 


1 





-4 


1 





~< 





1 


1 








1 











1 








-*] 


1 



G 2 = 



1 











1 








-4 i 








-x\ 


1 



(4.69) 



The Gauss transformation matrices may be applied to A, yielding an upper trian- 
gular matrix. This is illustrated by the following example. 

Example 4.4 Suppose that 



12 3 4 

-112 1 

2 13 

11 



(=A°). 
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We introduce matrices A , where A k — GkA k ~ l fork— 1, 2, . . . , n — 1, and finally 
U — A" -1 . Once again, A k is not the kth power of A, but rather denotes the kth 
matrix in a sequence of matrices. Now consider 



GiA v = 



10 
110 
10 
1 



12 3 4 

-112 1 

2 13 

11 



12 3 4 

3 5 5 

2 13 

11 



= A 1 



for which the x\ entries in the first column of G\ depend on the first column of 
A (i.e., of A) according to (4.65). Similarly 



G 2 A l 



for which the rf entries in the second column of G2 depend on the second column 
of A 1 , and also 



1 


ooo - 


"12 3 4" 




" 1 2 


3 


4 





1 


3 5 5 




3 


5 


5 






-§ 1 
1 _ 


2 13 
11 





_ 


7 
3 

1 


1 
3 

1 



GiA 1 



1 








1 


2 





1 








3 








1 














1 1 









4 




" 1 


2 


3 


4 


5 







3 


5 


5 


1 

3 










7 

3 


1 

3 


1 




. 








6 

7 



u 



for which the t. 3 entries in the third column of G3 depend on the third column of 
A 2 . We see that U is indeed upper triangular, and it is also nonsingular. We also 
see that 

U = G3G2G1 A. 

Since the product of lower triangular matrices is a lower triangular matrix, it is the 
case that L\ — G3G2G1 is lower triangular. Thus 

A = L~ l U. 

Since the inverse (if it exists) of a lower triangular matrix is also a lower triangular 
matrix, we can define L = L~^ , and so A = LU. Thus 

L = Gi G2 Go . 

From this example it appears that we need to do much work in order to find GT . 
However, this is not the case. It turns out that 



T k P T 

T e k-\- 



(4.70) 
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This is easy to confirm. From (4.68) and (4.70) 



G k G: 



U 
I - 



r k e T 



][/ 



r k eU] 



T k P T 

x e k-\ 



r kT 



k-lS'k-1- 



T k p T 
X e k-\ 



T k P T T k P T 
T e k-\ X e k-\ 



But from (4.66) and (4.67) e T k _ x x k = 0, so finally G k G~ l = /. 

To obtain r k from (4.65), we see that we must divide by x k -i- In our matrix 
factorization application of the Gauss transformation, we have seen (in Example 
4.4) that Xk-\ will be an element of A k . These elements are called pivots. It 
is apparent that the factorization procedure cannot work if a pivot is zero. The 
occurrence of zero-valued pivots is a common situation. A simple example of a 
matrix that cannot be factored with our algorithm is 



In this case 



G\A = 



and from (4.65) 



A = 



1 

l _ £i _ 
1 XQ 



1 

1 



Oio 

floo 



00. 



(4.71) 



(4.72) 



(4.73) 



This result implies that not all matrices possess an LU factorization. Let det(A) 
denote the determinant of A. We may state a general condition for the existence of 
the LU factorization: 



Theorem 4.1: Since A = [fl,,/];,/=o,...,n-i € R" x " we define the kth leading 
principle submatrix of A to be A k — [ajj];, ;=o,... ,k-\ € R* x * for k — 1,2, ... , n 
(so that A — A n , and A\ = [«oo] = aoo)- There exists a unit lower triangular matrix 
L and an upper triangular matrix U such that A — LU, provided that det(Ak) ^ 
for all k — 1, 2 . . . , n. Furthermore, with U — [uij] e R" x " we have det(Afc) = 

The proof is given in Golub and Van Loan [5]. It will not be considered here. For 
A in (4.71), we see that A\ — [0] = 0, so det(Ai) = 0. Thus, even though A -1 
exists, it does not possess an LU decomposition. It is also easy to verify that for 



4 



1 

8 1 
1 1 



although A exists, again A does not possess an LU decomposition. In this case 



we have det(A2) = det 



4 



= 0. Theorem 4. 1 leads to a test of positive 



definiteness according to the following theorem. 
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Theorem 4.2: Suppose R e R" x " with R = R T . Suppose that R = LDL T , 
where L is unit lower triangular, and D is a diagonal matrix (L — 
[li,j]i,j=o,...,n-i, D = [d/,/L-,/=o,...,n-i)- If rf/,i > for all i e Z„, then 7? > 0. 

Proof L is unit lower triangular, so for any y € R" there will be a unique 
x e R" such that 

y = L T x (y T =x T L) 

because L _1 exists. Thus, assuming D > 

71-1 

x r i?:t = x T LDL T x = y T Dy = ^y}d i4 > 

for all y # 0, since <i, j( - > for all i e Z„. In fact, ^f~o y?rfj,, = iff y; = for 
all i € Z„. Consequently, x T Rx > for all x ^ 0, and so immediately R > 0. 

We relate Z) in Theorem 4.2 to [/ in Theorem 4.1 according to t/ = DL T . If the 
LDL T decomposition of a matrix i? exists, then matrix D immediately tells us 
whether R is pd just by viewing the signs of the diagonal elements. 

We may define (as in Example 4.4) A k — [of ,], where k = 0, 1, . . . , n — 1 and 
A = A. Consequently 

r* = -5- = ^£^ (4.74) 

**"» «t-U-i 

for i = fe, k + 1, . . . , n — 1. This follows because Gt contains t. , and as observed 
in the example above, r* depends on the column indexed k — 1 in A* -1 . Thus, a 
pseudocode program for finding U can therefore be stated as follows: 

A°:=A; 

for /f := 1 to n - 1 do begin 
for / := k to n — 1 do begin 

r* := afr_i/a£l] t_n ; {This loop computes r''} 
end; 
/A A := G k A k ~^ ; { G^ contains zf via (4.64) } 
end; 
U:=A n - 1 ; 

We see that the pivots are aZZ\ t_j for A: = 1, 2, , . . , n — 1. Now 

U — G„-iG n -2 ■ ■ ■ G2G1 A. 

so 

A = G- l G- x ---G- l _ 2 G- l _ l U. (4.75) 
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Consequently, from (4.70), we obtain 

H-l 

L = (/ + x x el)(I + x 1 e\) •••(/ + r"" 1 ^) = / + £ r fc ej_ x . (4.76) 

To confirm the last equality of (4.76), consider defining L m = GJ~ •••G~ 1 for 
m = 1, . . . , n — 1. Assume that L m — I + Y1T=1 xke k-v wruc h is true for m — 1 
because L\ — G~[ = I + x x e^ . Consider L m+ \ = L m G~ +l , so 

L m+1 = (/ + fy e [_ 1 )(/ + r"'+ 1 ^) 



k=l 



=i+T,^ T k -i+* m+i ^+i: 



k „ T _1_ T m + l „ T _1_ X^T k „ T -r m + l „ T 



r e,_, + t e m + > r e,_,r e 



jt-l l e m- 



A:=l *:=! 



But ^_ 1 T m+1 = for k = 1, . . . , m from (4.66) and (4.67). Thus 

m m-\-\ 

L m+l =I + Y j r k e T k _ x + r m+l e T m =l+Y j r k eI-v 

k=\ k=\ 

Therefore, (4.76) is valid by mathematical induction. (A simpler example of a 
proof by induction appears in Appendix 3.B.) Because of (4.76), the previous 
pseudocode implicitly computes L as well as U. Thus, if no zero-valued pivots 
are encountered, the algorithm will terminate, having provided us with both L 
and U. [As an exercise, the reader should use (4.76) to find L in Example 4.4 
simply by looking at the appropriate entries of the matrices Gt\ that is, do not 
use L = Gj~ Gj G7 . Having found L by this means, confirm that LU — A.] We 
remark that (4.76) shows that L is unit lower triangular. 

It is worth mentioning that certain classes of matrix are guaranteed to possess 
an LU decomposition. Suppose that A e R" x " with A — A T and A > 0. Let v — 
[vo ■ ■ ■ Vk-\ • • • ] T ; then, if v ^ 0, we have v T Av > 0, but if A^ is the kth 

_ T n—k zeros 

leading principle submatrix of A, then 

v Av — u Akii > 

which holds for all k — 1,2, ... ,n. Consequently, A& > for all k, and so A^ 
exists for all k. Since A^ exists for all k, it follows that det(A^) ^ for all k. 
The conditions of Theorem 4.1 are met, and so A possesses an LU decomposi- 
tion. That is, all real-valued, symmetric positive definite matrices possess an LU 
decomposition. 
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We recall that the class of positive definite matrices is an important one since 
they have a direct association with least-squares approximation problems. This was 
demonstrated in Section 4.2. 

How many floating-point operations (flops) are needed by the algorithm for 
finding the LU decomposition of a matrix? Answering this question gives us an 
indication of the computational complexity of the algorithm. Neglecting multipli- 
cation by zero or by one, to compute A k — GkA k ~ l requires (« — k)(n — k + 1) 
multiplications, and the same number of additions. This follows from considering 
the product GkA k ~ l with the factors partitioned into submatrices according to 



G k 



h 
T k 




h-k 



,/t-l 



,k-l 
1 00 





i oi 



(4.77) 



where It is a k x k identity matrix, 7* is (« — k) x k and is zero-valued except for 



k-\ 



its last column, which contains — r [see (4.64)]. Similarly, A 00 is (k — 1) x 



(k - 1), A* is (k - 1) x (« 



.k-i 



1), and A,, ' is (n - k + 1) x (n 



1). 



From the pseudocode, we see that we need ~YTk=\ ( n ~ ^) division operations. 
Operation A k — GtA k ~ x is executed for k = 1 to n — 1, so the total number of 
operations is: 

M-l 



2_\( n — k)(n — k + \) multiplications 

k=\ 

n-\ 

2__S n — k)(n — k + \) additions 

k=\ 

n-\ 

J> - k) 



divisions 



We now recognize that 



iV 



k=\ 



N(N+1) ^ 2 _ N(N + l)(2N + 1) 



I> 2 
jt=i 



(4.78) 



where the second summation identity was proven in Appendix 3.B. The first sum- 
mation identity may be proved in a similar manner. Therefore 



n-l 



H-l 



k=\ 



J2( n - k )( n ~ k + 1) = J^[« 2 + n - (2n + 1)£ + k 2 ] 

k=\ 

n— 
= {n- 1)(« 2 + n) - (2n + l)J2 k + J2 k2 



n—\ n — \ 



k=l k=l 
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1 , 1 

= -n 3 «, (4.79a) 

3 3 

«— 1 n— 1 . -. 

Y^(n-k) = n(n- l)-^k = -n 2 - -n. (4.79b) 

k=l k=\ 

So-called asymptotic complexity measures are denned using 

Definition 4.2: Big O We say that fin) = 0(g(n)) if there is a < c < oo, 
and an N € N (N < oo) such that 

fin) < cgin) 

for all n > N. 

Our algorithm needs a total of fin) — 2(|« 3 — |n) + jn 2 — jn — |w 3 + jn 2 — 
In flops. We may say that 0(n 3 ) operations (flops) are needed (so here gin) = 
n 3 ). We may read 0(« 3 ) as "order n-cubed," so order n-cubed operations are 
needed. If one operation takes one unit of time on a computing machine we say 
the asymptotic time complexity of the algorithm is 0(w 3 ). Parameter n (matrix 
order) is the size of the problem. We might also say that the time complexity 
of the algorithm is cubic in the size of the problem since the number of oper- 
ations fin) is a cubic polynomial in n. But we caution the reader about flop 
counting: 

"Flop counting is a necessarily crude approach to the measuring of program efficiency 
since it ignores subscripting, memory traffic, and the countless other overheads asso- 
ciated with program execution. We must not infer too much from a comparison of 
flops counts. . . . Flop counting is just a 'quick and dirty' accounting method that 
captures only one of several dimensions of the efficiency issue." 

— Golub and Van Loan [5, p. 20] 

Asymptotic complexity measures allow us to talk about algorithmic resource 
demands without getting bogged down in detailed expressions for computing time, 
memory requirements, and other variables. However, the comment by Golub and 
Van Loan above may clearly be extended to asymptotic measures. 

Suppose that A is LC/-factorable, and that we know L and U . Suppose that we 
wish to solve Ax — y. Thus 

LUx = y, (4.80) 

and define Ux — z, so we begin by considering 

Lz = y. (4.81) 
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In expanded form this becomes 



4o 











ho 


lu 








ho 


hi 


'22 






'«-i,o 4-1,1 '«-i,2 ••• 4-l,«-l 





zo 




yo 




Z\ 




yi 




Zl 


= 


yi 




Z n -\ 




y n -\ 



(4.82) 



Since L ' exists, solving (4.81) is easy using forward elimination {forward substi- 
tution). Specifically, from (4.82) 



zo 



yo 
4,o 



z\ = - — [y\ -zo4,o] 
'1,1 



Z2 = " Ij2 - Zoh,0 - Zl/2,l] 

'2,2 



Thus, in general 



Zn-l = 



'« — l,n- 



Z* = 



lk„ 



n-2 

y n -\ — / ,Zkln-l,k 

k=Q 



k-l 

yk ~ ^ Zjlk.i 
i=0 



(4.83) 



for fc = 1, 2, ...,« — 1 with zo = yo/4,0- Since we now know z, we may solve 
[/x = z by backward substitution. To see this, express the problem in expanded 
form: 



«0,0 "0,1 
Ml 1 








M 0,n-2 M0,h-1 

"l,n-2 «l,n-l 



tin— 2,w— 2 W«— 2, n — 1 
M„_i„_i 





x 




zo 




XI 




Zl 




*n-2 




Zn-2 




_ ■*»-! 




Zn-l 



From (4.84), we obtain 



Xn—l 



Xn—2 



Zn-l 
Mn—\,n—\ 

1 

Un—2,n—2 



-YZn-2 — X„-lU n -2,n-l] 



(4.84) 
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X n -3 



Mn—3,n—3 



~iZn— 3 x n — \Un—?>,n — { Xn—2M-n—3,n—2] 



Xq = 



"0,0 



zo 



n-\ 

/ ,XkUQ,k 
k=\ 



In general 



** 



1 
Uk,k 



Zk 



^2 XiUk,i 



i=k+\ 



(4.85) 



for k — n — 2, . . . ,0 with x n -\ — z n -\/u n -i,n-\- The forward-substitution and 
backward-substitution algorithms that we have just derived have an asymptotic 
time complexity of 0(n 2 ). The reader should confirm this as an exercise. This 
result suggests that most of the computational effort needed to solve for x in 
Ax = y lies in the LU decomposition stage. 

So far we have said nothing about the performance of our linear system solution 
method with respect to finite precision arithmetic effects (i.e., rounding error). 
Before considering this matter, we make a few remarks regarding the stability of 
our method. We have noted that the LU decomposition algorithm will fail if a zero- 
valued pivot is encountered. This can happen even if Ax — y has a solution and A 
is well-conditioned. In other words, our algorithm is actually unstable since we can 
input numerically well-posed problems that cause it to fail. This does not necessarily 
mean that our algorithm should be totally rejected. For example, we have shown 
that positive definite matrices will never result in a zero-valued pivot. Furthermore, 
if A > 0, and it is well-conditioned, then it can be shown that an accurate answer 
will be provided by the algorithm despite its faults. Nonetheless, the problem of 
failure due to encountering a zero-valued pivot needs to be addressed. Also, what 
happens if a pivot is not exactly zero, but is close to zero? We might expect that 
this can result in a computed solution x that differs greatly from the mathematically 
exact solution x, especially where rounding error is involved, even if A is well- 
conditioned. 

Recall (2.15) from Chapter 2, 



fl[x op y] = (x op y)(l + e) 



(4.86) 



for which |e| < 2 '. If we store A in a floating-point machine, then, because of 
the necessity to quantize we are really storing the elements 



[fl[A]]ij=fl[a i j] = a i j(l + e iJ ) 



(4.87) 
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with |e,-j| < 2~'. Suppose now that A, 5 e R mx "; we then define 6 

|A| = [|a u |]eR mxn , (4.88) 

and by B < A, we mean bij < aij for all i and j. So we may express (4.87) more 
compactly as 

\fl[A]-A\<u\A\, (4.89) 

where u — 2~' , since |e,- ;| < 2~' . 

Forsythe and Moler [4, pp. 104-105] show that the computed solution z to 
Lz — y [recall (4.81)] as obtained by forward substitution is actually the exact 
solution to a perturbed lower triangular system 

(L + 8L)z = y, (4.90) 

where SL is a lower triangular perturbation matrix, and where 

|<5L| < 1.01nw|L|. (4.91) 

A very similar bound exists for the problem of solving Ux — z by backward- 
substitution. We will not derive these bounds, but will simply mention that the 
derivation involves working with a bound similar to (2.39) in Chapter 2. From 
(4.91) we have relative perturbations -rMr < l.Olnw. It is apparent that since u 
is typically quite tiny, unless n (matrix order) is quite huge, these relative pertur- 
bations will not be significant. In other words, forward substitution and backward 
substitution are very stable procedures that are quite resistant to the effects of 
rounding errors. Thus, any difficulties with our linear system solution procedure in 
terms of a rounding error likely involve only the LU factorization stage. 

The rounding error analysis for our Gaussian elimination algorithm is even more 
involved than the effort required to obtain (4.91), so again we will content ourselves 
with citing the main result without proof. We cite Theorem 3.3.1 in Golub and Van 
Loan [5] as follows. 

Theorem 4.3: Assume that A is an n x n matrix of floating-point numbers. 
If no zero-valued pivots are encountered during the execution of the Gaussian 
elimination algorithm for which A is the input, then the computed triangular factors 
(here denoted L and U) satisfy 

LU = A + 8A (4.92a) 

such that 

\SA\ < 3(« - 1)m(|A| + \L\\U\) + 0(u 2 ). (4.92b) 

There is some danger in confusing this with the determinant. That is, some people use \A\ to denote 
the determinant of A. We will avoid this here by sticking with det(A) as the notation for determinant 
of A. 
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In this theorem the term 0(u 2 ) denotes a part of the error term dependent on u 2 . 
This is quite small as u 2 — 2~ 2t (rounding assumed), and so may be practically 
disregarded. The term arises in the work of Golub and Van Loan [5] because those 
authors prefer to work with slightly looser bounding results than are to be found 
in the volume by Forsythe and Moler [4]. The bound in (4.92b) gives us cause for 
concern. The perturbation matrix S A may not be small. This is because |L||L/| can 
be quite large. An example of this would be 



A = 



1 4 1 

2 8.001 1 
-1 1 



for which 



1 

2 1 
-1000 1 



U 



1 4 1 

0.001 -1 
-999 



This has happened because 



A 1 = 



1 


4 


1 





0.001 


-1 





-1 


1 



which has a\ j = 0.001. This is a small pivot and is ultimately responsible for 
giving us "big" triangular factors. Clearly, the smaller the pivot the bigger the 
potential problem. Golub and Van Loan's [5] Theorem 3.3.2 (which we will not 
repeat here) goes on to demonstrate that the errors in the computed triangular 
factors can adversely affect the solution to Ax = L Ux — y as obtained by forward 
substitution and backward substitution. Thus, if we use the computed solutions L 
and U in LUx = y, then the computed solution x may not be close to x. 

How may our Gaussian elimination LU factorization algorithm be modified to 
make it more stable? The standard solution is to employ partial pivoting. We do not 
consider the method in detail here, but illustrate it with a simple example (Example 
4.5). Essentially, before applying a Gauss transformation Gt, the rows of matrix 
A k ~ l are permuted (i.e., exchanged) in such a manner as to make the pivot as 
large as possible while simultaneously ensuring that A k is as close to being upper 
triangular as possible. Permutation operations have a matrix description, and such 
matrices may be denoted by P^. We remark that P7 — Pi . 



Example 4.5 Suppose that 



A = 



1 


4 


1 


2 


8 


1 





-1 


1 



= A h 
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Thus 



G1P1A 



o 



10 0" 




"010" 




" 1 


4 


1 " 




\ 1 




1 




2 


8 


1 




1 _ 




_ 1 _ 




_ 


-1 


1 




10 0" 




"2 8 1" 




" 2 


8 


1 


\ 1 




1 4 1 


= 








l 

2 


1 _ 




-1 1 






_ 


-1 


1 



and 





"10 




"10 0" 




"2 8 1 


G2P2A 1 = 


1 
_ 1 _ 




1 
_ 1 _ 




1 

_ -1 1 




"10 0" 




"2 8 1" 




= 


1 




-1 1 


= A 2 




1 




.0 1 







for which U — A 2 . We see that P 2 interchanges rows 2 and 3 rather than 1 and 
2 because to do otherwise would ruin the upper triangular structure we seek. It is 
apparent that 

G 2 P 2 GiPiA = U, 



so that 



for which 



A=P- 1 G~ 1 P- 1 G^U, 



=L 



r 1 



1 



1 
1 



This matrix is manifestly not lower triangular. Thus, our use of partial pivoting to 
achieve algorithmic stability has been purchased at the expense of some loss of 
structure (although Theorem 3.4.1 in Ref. 5 shows how to recover much of what 
is lost. 7 ) Also, permutations involve moving data around in the computer, and this 
is a potentially significant cost. But these prices are usually worth paying. 

In general, the Gaussian elimination with partial pivoting algorithm generates 



and it turns out that 



G n _ 1 P n _ 1 -G 2 P 2 G 1 P 1 A = U, 



P„_l ■ ■ ■ P 2 PiA = LU 



for which L is unit lower triangular, and U is upper triangular. The expression for L in terms of the 
factors Gk is messy, and so we omit it. The interested reader can see pp. 112-113 of Ref. 5 for details. 
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It is worth mentioning that the need to trade off algorithm speed in favor of sta- 
bility is common in numerical computing; that is, fast algorithms often have stability 
problems. Much of numerical computing is about creating the fastest possible stable 
algorithms. This is a notoriously challenging engineering problem. 

A much more detailed account of Gaussian elimination with partial pivoting 
appears in Golub and Van Loan [5, pp. 108-1 16]. This matter will not be discussed 
further in this book. 



4.6 LEAST-SQUARES PROBLEMS AND QR DECOMPOSITION 

In this section we consider the QR decomposition of A € R mx " for which m > n, 
and A is of full rank [i.e., rank (A) = «]. Full rank in this sense means that the 
columns of A are linearly independent. The QR decomposition of A is 

A = QR, (4.93) 

where Q e R mxm is an orthogonal matrix [i.e., Q T Q — QQ T — I (identity 
matrix)], and R e R mx " is upper triangular in the following sense: 



R = 



0.0 




ro,i • 


ro,n-l 

r\, n -\ 






• 
• 


fn — \,n — l 

















n 
o 



(4.94) 



Here TZ e R" x " is a square upper triangular matrix and is nonsingular because A 
is full rank. The bottom block of zeros in R of (4.94) is (m — n) x n. 

It should be immediately apparent that the existence of a QR decomposition 
for A makes it quite easy to solve for x in Ax — y, if A -1 exists (which implies 
that in this special case A is square). Thus, Ax — QRx — y, and so Rx — Q T y. 
The upper triangular linear system Rx — Q T y may be readily solved by backward 
substitution (recall the previous section). 

The case where m > n is important because it arises in overdetermined least- 
squares approximation problems. We illustrate with the following example based on 
a real-world problem. 8 Figure 4. 1 is a plot of some simulated body core temperature 

This example is from the problem of estimating the circadian rhythm parameters of human patients 
who have sustained head injuries. The estimates are obtained by the suitable processing of various 
physiological data sets (e.g., body core temperature, heart rate, blood pressure). The nature of the injury 
has made the patients' rhythms deviate from the nominal 24-h cycle. Correct estimation of rhythm 
parameters can lead to improved clinical treatment because of improved timing in the administering of 
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O 



Q. 
E 
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Illustration of least-squares fitting 
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Figure 4.1 Simulated human patient temperature data to illustrate overdetermined least- 
squares model parameter estimation. Here we have N = 1000 samples f n (the dots), for 
T s = 300 (seconds), T = 24 (hours), a = 2 x 10" 7o C/s, b = 37°C, and c = 0.1°C. The 
solution to (4.103) is a = 2.0582 x 10" 7o C/s, b = 36.9999°C, c = 0.1012°C. 



measurements 

has three components 



from a human patient (this is the noisy data with trend). The data 



1. A sinusoidal component 

2. Random noise. 

3. A linear trend. 

Our problem is to estimate the parameters of the sinusoid (i.e., the amplitude, 
period, and phase), which represents the patient's circadian rhythm. In other words, 
the noise and trend are undesirable and so are to be, in effect, removed from the 
desired sinusoidal signal component. Here we will content ourselves with estimating 
only the amplitude of the sinusoid. The problem of estimating the remaining param- 
eters is tougher. Methods to estimate the remaining parameters will be considered 
later. (This is a nonlinear optimization problem.) 

We assume the model for the data in Fig. 4.1 is the analog signal 



2n 
f(t) = at + b + c sin I — t 



lit). 



(4.95) 



Here the first two terms model the trend (assumed to be a straight line), the third 
term is the desired sinusoidal signal component, and r](t) is a random noise com- 
ponent. We only possess samples of the signal /„ = f(nT s ) (i.e., t — nT s ), for 
n — 0, I, . . . , N — 1 , where T s is the sampling period of the data collection system. 

medication. We emphasize that the model in (4.95) is grossly oversimplified. Indeed, a better model is 
to replace term at + b with subharmonic, and harmonic terms of sin I ^j-t\. A harmonic term is one 

of frequency -S-n, while a subharmonic has frequency -S--- Cosine terms should also be included in 
the improved model. 
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We assume that we know T which is the period of the patient's circadian rhythm. 
Our model also implicitly assumes knowledge of the phase of the sinusoid, too. 
These are very artificial assumptions since in practice these are the most important 
parameters we are trying to estimate, and they are never known in advance. How- 
ever, our present circumstances demand simplification. Our estimate of /„ may be 
defined by 

f n =aT s n + b + csm[ — nT s . (4.96) 



This is a sampled version of the analog model, except the noise term has been 
deleted. 

We may estimate the unknown model parameters a, b, c by employing the same 
basic strategy we used in Section 4.2, specifically, a least-squares approach. Thus, 
defining x — [a b c] T (vector of unknown parameters), we strive to minimize 



JV— 1 



AT— 1 



v(*) = j>„ 2 = £[/„- ;„] 2 



(4.97) 



n=0 



n=0 



with respect to x. Using matrix/vector notation was very helpful in Section 4.2, 
and it remains so here. Define 



T s ii 1 sin | — T s n 



(4.98) 



Thus 



fn ~ V n X. 



(4.99) 



We may define the error vector e — [eo e\ • • 
[/o /l • • • In-iV > and the matrix of basis vectors 



eN-\] T , data vector f 



A = 



T 
..T 



L "N-l J 



eR 



JVx3 



(4.100) 



Consequently, via (4.99) 



e — f — Ax. 



(4.101) 



Obviously, we would like to have e = 0, which implies the desire to solve Ax — f. 
If we have N — 3 and A -1 exists, then we may uniquely solve for x given any 
/. However, in practice, N >> 3, so our linear system is overdetermined. Thus, 
no unique solution is possible. We have no option but to select x to minimize e in 
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some sense. Once again, previous experience from Section 4.2 says least-squares 
is a viable choice. Thus, since IHI2 = e T e — *}2 n =o e n> we consider 

V(x) = e T e= f T f -2x T A T f +x T A T Ax (4.102) 



[which is a more compact version of (4.97)]. This is yet another quadratic form 
[recall (4.8)]. We see that P e R 3x3 , and g e R 3 . In our problem A is full rank so 
from the results in Section 4.2 we see that P > 0. Naturally, from the discussions 
of Sections 4.3 and 4.4, the conditioning of P is a concern. Here it turns out that 
because P is of low order (largely because we are interested only in estimating 
three parameters) it typically has a low condition number. However, as the order 
of P rises, the conditioning of P usually rapidly worsens; that is, ill conditioning 
tends to be a severe problem when the number of parameters to be estimated rises. 
From Section 4.2 we know that the optimum choice for x, denoted x, is obtained 
by solving the linear system 

Px = g. (4.103) 

The model curve of Fig. 4.1 (solid line) is the curve obtained using x in (4.96). 
Thus, since x — [a b c] T , we plot /„ for a,b,c in place of a,b,c in (4.96). 
Equation (4.103) can be written as 

A T Ax = A T f. (4.104) 

This is just the overdetermined linear system Ax — f multiplied on the left (i.e., 
premultiplied) by A T . The system (4.104) is often referred to in the literature as 
the normal equations. 

How is the previous applications example relevant to the problem of QR fac- 
torizing A as in Eq. (4.93)? To answer this, we need to consider the condition 
numbers of A, and of P — A T A, and to see how orthogonal matrices Q facili- 
tate the solution of overdetermined least-squares problems. We will then move on 
to the problem of how to practically compute the QR factorization of a full-rank 
matrix. We will consider the issue of conditioning first since this is a justification 
for considering QR factorization methods as opposed to the linear system solution 
methods of the previous section. 

Singular values were mentioned in Section 4.4 as being relevant to the problem 
of computing spectral norms, and so of computing K2(A). Now we need to consider 
the consequences of 

Theorem 4.4: Singular Value Decomposition (SVD) Suppose A € R mxn ; 
then there exist orthogonal matrices 

U = [«0K1 ' ' ' «m-l] € R mxm , V = [W0W1 • • • V n -l] € R BXB 

such that 

E = U T AV = diag (ct , o u . . . , o p _{) e R mx ", p = min{m, «}, (4.105) 

where oq > o\ > . . . > er„-i > 0. 
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An outline proof appears in Ref. 5 (p. 71) and is omitted here. The nota- 
tion diag(tro, • ■ ■ , o>-i) means a diagonal matrix with main diagonal elements 
ao, . . . , o p -\. For example, if m — 3, n — 2, then p — 2, and 



U T AV 



CTQ 
(7! 





but if m = 2, n = 3, then again p — 2, but now 



U T AV 



o-o 
ori 



The numbers er, are called singular values. Vector ut is the ith left singular vector, 
and vi is the ith right singular vector. The following notation is helpful: 

cr r (A) = the ith singular value of A (i e Z p ). 
cr max (A) = the biggest singular value of A. 
&mm(A) = the smallest singular value of A. 

We observe that because AV — WE, and A T U — VE T we have, respectively 

Avj — OiUi, A Uj — OiV{ (4.106) 

for i e Z p . Singular values give matrix 2-norms; as noted in the following theorem. 

Theorem 4.5: 

l|A|| 2 = O-o = 0-max(A). 

Proof Recall the result (4.37). From (4.105) A = UEV T so 



and 



\Ax\\l =x T A T Ax, 



P -\ 



A T A = VE^Ey 7 = J^aj L v i vf € R" x ". 
For any x e R" there exist di such that 

n-l 
X = 'Y^diVi 



(4.107) 



(4.108) 



;=o 



(because V is orthogonal so its column vectors form an orthogonal basis for R"). 
Thus 



-i»-i 



-ill = xTx = J2J2 did J v ^ v J = 12 d ? 



(4.109) 



i=o ;=o 



i=0 
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(via vJvj — S{-j). Now 



x 



P -\ P -\ 

T A T Ax = Y]<T?(x T Vi)(v?x) = J2 a ^ x ' ^ >2 ' (4.110) 

i=0 (=0 



but 

(x, vi) = {Y^djVj, V{ J = ^djjvj, vj) = <i,-. (4.111) 

\ .7 ' .7 

Using (4.111) in (4.110), we obtain 

M-l 

;=o 

,2 — a f„„ ; ^ „ r \ir_ ™ ^; m ;,a II A -*■! |2 



for which it is understood that <r = for i > p — 1. We maximize 1 1 Ax\ \^ subject 



to constraint ||x||2 = 1, which means employing Lagrange multipliers; that is, we 



maximize 

n— 1 /« — 1 



L(d) = J2 a i Ld ?- x IZ^ 2-1 ' (4.113) 



(=0 \(=0 

where d = [do d\ ■ ■ ■ d n -\] T . Thus 

8L(d) 2 









ddj 


"7 z - /v< 


*■} — 


or 






ajdj 


= A.dj . 




From (4.114) 


into 


(4.112) 




n-\ 










\\Ax\\ 2 2 = 


^E d ? 


= a 



(4.114) 



(4.115) 



i'=0 



for which we have used the fact that \\x\\\ = 1 in (4.109). From (4.114) X is the 
eigenvalue of a diagonal matrix containing of. Consequently, k is maximized for 

X — Oq. Therefore, ||A||2 = &o- 

Suppose that 

OQ > • • • > O r -\ > O r — ■ ■ ■ — Op-i — 0, (4.116) 

then 

rank(A) = r. (4.117) 
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Thus, the SVD of A can tell us the rank of A. In our overdetermined least-squares 
problem we have m > n and A is assumed to be of full-rank. This implies that 
r — n. Also, p — n. Thus, all singular values of a full-rank matrix are bigger than 
zero. Now suppose that A -1 exists. From (4.105) A -1 = VT,~ l U T . Immediately, 
l|A- 1 || 2 = l/crmin(A). Hence 

^2(A) = ||A|| 2 ||A- 1 || 2 = gmax( ^ ) . (4.118) 

Cmin(A) 

Thus, a large singular value spread is associated with matrix ill conditioning. [Recall 
(4.61) and the related discussion.] As remarked on p. 223 of Ref. 5, Eq. (4.118) 
can be extended to cover full-rank rectangular matrices with m > n: 

A€R mx ",rank(A) = «^/C2(A)= °" max(A) . (4.119) 

CTminCA) 

This also holds for the transpose of A because A T — VT, T U T , so A T has the 
same singular values as A. Thus, *r 2 (A r ) = K2(A), Golub and Van Loan [5, p. 225] 
claim (without formal proof) that /c 2 (A r A) = [/c 2 (A)] 2 . In other words, if the linear 
system Ax — f is ill-conditioned, then A T Ax — A T f is even more ill-conditioned. 
The condition number of the latter system is the square of that of the former system. 
More information on the conditioning of rectangular matrices is to be found in 
Appendix 4.B. This includes justification that Ar 2 (A r A) = [k 2 (A)] 2 . 

A popular approach toward solving the normal equations A T Ax — A T f is based 
on Cholesky decomposition 

Theorem 4.6: Cholesky Decomposition If R e R" xn is symmetric and pos- 
itive definite, then there exists a unique lower triangular matrix L e R" x " with 
positive diagonal entries such that R — LL T . This is the Cholesky decomposition 
(factorization) of R. 

Algorithms to find this decomposition appear in Chapter 4 of Ref. 5. We do not 
consider them except to note that if they are used, then the computed solution to 
A T Ax = A T f, which we denote by x, may satisfy 

ll *~* lh ~u[K 2 (A)f, (4.120) 



ll*||2 

where u is as in (4.89). Thus, this method of linear system solution is potentially 
highly susceptible to errors due to ill-conditioned problems. On the other hand, 
Cholesky approaches are computationally efficient in that they require about n 3 /3 
flops (Floating-point operations). Clearly, Gaussian elimination may be employed 
to solve the normal equations as well, but we recall that Gaussian elimination 
needed about 2n 3 /3 flops. Gaussian elimination is less efficient because it does 
not account for symmetry in matrix R. Note that these counts do not take into 
consideration the number of flops needed to determine A T A and A T f, and do 
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not account for the number of flops needed by the forward/backward substitution 
steps. However, the comparison between Cholesky decomposition and Gaussian 
elimination is reasonably fair because these other steps are essentially the same for 
both approaches. 



Recall that \\e\\\ — \\Ax — f\\\. Thus, for orthogonal matrix Q 
\\Q T e\\l = [Q T e] T Q T e = e T QQ T e = e T e = ||e|||. 



(4.121) 



Thus, the 2-norm is invariant to orthogonal transformations. This is one of the 
more important properties of 2-norms. Now consider 



\\e\\ 2 2 = \\Q T Ax-Q T f\\ 2 2 . 



(4.122) 



Suppose that 



e r / = 



./■" 



(4.123) 



for which /" € R", and /' € R m_ ". Thus, from (4.94) and Q T A = R, we obtain 



Q T Ax - Q T f = 



Kx- f u 

-f 



implying that 
Immediately, we see that 



kill = lift* - /"111 + ll/'lll. 



Kx = f u . 



(4.124) 

(4.125) 
(4.126) 



The least-squares optimal solution x is therefore found by backward substitution. 
Equally clearly, we see that 



min||e||i = ||/'||i 



2 

Pls- 



(4.127) 



This is the minimum error energy. Quantity p^ s is also called the minimum sum 
of squares, and e is called the residual [5]. It is easy to verify that /C2(G) = 1 
(Q is orthogonal). In other words, orthogonal matrices are perfectly conditioned. 
This means that the operation Q T A will not result in a matrix that is not as well 
conditioned as A. This in turn suggests that solving our least-squares problem 
using QR decomposition might be numerically more reliable than working with 
the normal equations. As explained on p. 230 of Ref. 5, this is not necessarily 
always true, but it is nevertheless a good reason to contemplate QR approaches to 
solving least-squares problems. 9 

y If the residual is big and the problem is ill-conditioned, then neither QR nor normal equation methods 
may give an accurate answer. However, QR approaches may be more accurate for small residuals in 
ill-conditioned problems than normal equation approaches. 
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How may we compute Ql There are three major approaches: 

1. Gram-Schmidt algorithms 

2. Givens rotation algorithms 

3. Householder transformation algorithms 

We will consider only Householder transformations. 

We begin by a review of how vectors are projected onto vectors. Recall the 
law of cosines from trigonometry in reference to Fig. 4.2a. Assume that x, y e R". 
Suppose that \\x — y\\2 = a, \\x\\i — b, and that \\yW2 — c. Therefore, where 9 is 
the angle between x and y (0 < 9 < n radians) 



b 2 + c 2 - 2bccos9, 



(4.128) 



or in terms of the vectors x and y, Eq. (4.128) becomes 

\\x-y\\ 2 2 = \\x\\l + \\y\\ 2 2 -2\\x\\ 2 \\y\\2Cos$. 
In terms of inner products, this becomes 

(x-y,x-y) = (x, x) + (y, y) - 2[(x, x)]^ 2 [(y, y)} 1 ' 2 cos9, 
which reduces to 

(x,y) = l(x,x)] 1/2 [(y,y)] l/2 c 0& 9, 



(4.129) 



(x,y) 



\x\\2\\y\\2COSt 



(4.130) 



(a) 





^y 



Figure 4.2 Illustration of the law of cosines (a) and the projection of vector x onto vector 

y (b). 
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Now consider Fig. 4.2b. Vector P y x is the projection ofx onto y, where P y denotes 
the projection operator that projects x onto y. It is immediately apparent that 



\PyX\\2 = ||x|| 2 COS( 



(4.131) 



This is the Euclidean length of P y x. The unit vector in the direction of y is y/\ \y\ | 2 . 

Therefore 

llxll? COS0 

-y. (4.132) 



PyX 



\\y\h 



But from (4.130) this becomes 



(x,y) 

P y X = Try. 

\\y\\\ 



(4.133) 



Since (x, y) — x y — y x, we see that 



T 

y x 
P y x = ^—ry = 



\\y\\\ 



yy 



(4.134) 



In (4.134) yy € R" x ", so the operator P y has the matrix representation 

1 



Py = 



ml yy 



In Fig. 4.2b we see that z — x — P y x, and that 



Z = (I ~ Py)X 



1 



\\y\\\ 



yy 



X, 



which is the component of x that is orthogonal to y. We observe that 



1 



IMI 



T T 

jyy yy 



y\\y\\ 2 2 y T 
\\y\\i 



^r 3 



Py 



(4.135) 



(4.136) 



(4.137) 



If A 2 — A, we say that matrix A is idempotent. Thus, projection operators are 
idempotent. Also, Pf = P y so projection operators are also symmetric. 

In Fig. 4.3, x, y, z € R", and y T z — 0. Define the Householder transformation 
matrix 



H = I 



, yy 

'\\y\\i 



(4.138) 



We see that H — I — 2P y [via (4.135)]. Hence Hx is as shown in Fig. 4.3; that 
is, the Householder transformation finds the reflection of vector x with respect to 
vector z, and z -L y (Definition 1.6 of Chapter 1). Recall the unit vector e, € R" 

ei = [0---oio---0] r , 
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Figure 4.3 Geometric interpretation of the Householder transformation operator H . Note 
that z T y = 0. 



so eo = [10 • • • 0] r . Suppose that we want Hx — aeo for some a e R with a ^ 0; 
that is, we wish to design H to annihilate all elements of x except for the top 
element. Let y — x + aeo; then 



T T T T 

y x — (x + ae Q )x — x x + cexo 
(as x — [xq x\ ■ ■ ■ x n -\\ T ), and 

||y|| 2 = (x + ae Q )(x + aeo) — x x + 2axo + a 
Therefore 

T T 

Hx — x — 2- — —x = x — 2 -y 



\\y\\ 2 2 



\\y\\V 



so from (4.139), this becomes 



Hx = x — 2 



x — 2 



(x x + axQ)(x + aeo) 

iiyiil 

(x x + ajto)-^ + &(y x)eo 



1 -2- 



x x + ajto 



x — 2a 



T 
y x 



x T x + 2ax + a 2 ] \\y\\i 

To force the first term to zero, we require 

x x + 2axQ + a — 2(x x + oixq) — 0, 
which implies that a 2 — x T x, or in other words, we need 

« = ±l|x|| 2 . 



r^o- 



(4.139a) 



(4.139b) 



(4.140) 



(4.141) 
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Consequently, we select y — x ± ||x||2<?o- I n this case 



y T x x T x + axQ 
Hx - —2a T-e = —2a — T e 



I l^lll 



x T x + 2a.ro + « 2 



a + axo 

= ~ 2q! t 7 , -, g o = -ae = TlMbeo. 

LoL L + 2axo 



(4.142) 



so Hx = aeo for a — —a if y — x + aeo with a = ±||x||2- 

Example 4.6 Suppose x — [4 3 0] r , so ||jc||2 = 5. Choose a = 5. Thus 

y = [9 3 Of, and 



H = 1-2 



yy _ j_ 

'y T y ~ 45 



-36 -27 

-27 36 

45 



We see that 




















1 
Hx = — 

45 


' -36 


-27 


" 




" 4 " 


1 

= 45 


" -225 " 




" -5 " 


-27 



36 





45 




3 







= 






so Hx — —aeo 


. 



















The Householder transformation is designed to annihilate elements of vectors. But 
in contrast with the Gauss transformations of Section 4.5, Householder matrices 
are orthogonal. To see this observe that 



H T H = 



1-2 



yy 

yTy 



1-2 



yy 
y T yj 



T T T 

y T y iy T yf 

T T 

= 7-4^+4^ = 7. 

y y y y 



Thus, no matter how we select y, K2(H) = 1. Householder matrices are therefore 
perfectly conditioned. 

To obtain R in (4.94), we define 



H k 



4-i 








€R" 



(4.143) 
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where k — 1,2, ... ,n, h-i is an order k — 1 identity matrix, Hk is an order m — 
k + 1 Householder transformation matrix. We design Hk to annihilate elements k 
to m — 1 of column A: — 1 in A k ~ l , where A — A, and 



A k = H k A k ~\ 



(4.144) 



so A" = R (in (4.94)). Much as in Section 4.5, we have A k = [af ,.] € R mx ", and 
we assume m > n. 

Example 4.7 Suppose 



'oo 
i° 

MO 
'20 
-'30 



-*01 
,,0 



*21 



-*31 



*02 

1° 
-M2 

,.0 
'22 

'32 



eR 



4x3 



and so therefore 



H 1 A V 



A 2 = H 2 A X = 



X X X X 

X X X X 

X X X X 

X X X X 



10 

x x x 

x x x 

x x x 



" a 
"oo 

a 

"10 

a 

"20 

"30 
fl 1 

"oo 








"01 

a 

"21 

fl° 
"31 

'01 

Ml 

3 21 

'31 



"02 

a 

"12 
Q 22 

Q 32 

7 1 

'02 

2 1 

M2 

3 22 
'32 



r fl 1 

"00 







a 1 
"oo 







"01 

"11 

a 21 

"31 

a 1 

"oi 

8 n 





a 1 "" 

"02 

"12 

a 22 

"32 

7 1 

'02 

2 2 

Z 2 
-'22 

2 2 
^32 J 



and 



A 3 = H 3 A 2 



10 

10 

x x 

x x 



fl 00 







"01 

°n 







"02 

a 2 

"l2 
3 22 





r a 1 
"oo 





= R. 



"01 

a ii 




2 1 "" 
"02 

2 2 
l 2 

"22 

2 2 
-'32 
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The x signs denote the Householder matrix elements that are not specified. This 
example is intended only to show the general pattern of elements in the matrices. 

Define 

** = [4-l.k-l 4U-1 • • • 4-U-lf € Rm "" +1 ' ( 4 - 145 ) 

so if x k = [x k x\ ■■■ x k m _ k f then x k = a\~l_ xk _ v and so 

y k (y k ) T 

H k = l m _ k+l -2 y -^- k , (4.146) 

where y k — x k ± | |jc*| |2^q, and e\ = [1 • • • 0] r € R m_ * +1 . A pseudocode anal- 
ogous to that for Gaussian elimination (recall Section 4.5) is as follows: 

A°:=A; 

for k := 1 to n do begin 

for / := to m — k do begin 

x k := av~, _., fc1 ; {This loop makes x^} 
end; 
y k :=x' t + sign(x*)||x' ( ||2eg; 
A k := H k A k ~ 1 ; { H k contains H k via (4.146) } 
end; 
R:=A n ; 

From (4.143) H. H k — I m because H. H k — I m - k +\, and of course /Jj/jt-i = 
4_i; that is, H^ is orthogonal for all k. Since 

R = A"=H n H n - 1 ---H 2 H l A, (4.147) 

we have 

A = Hi Hi ■■■ Hi R. (4. 148) 

=Q 

Thus, the pseudocode above implicitly computes Q because it creates the orthog- 
onal factors Hk- 

In the pseudocode we see that 

y*=;c* + sign(*J)||**||2e§. (4.149) 

Recall from (4.141) that a = ±||x||2, so we must choose the sign of a. It is best that 
a — sign(*o)||x||2, where sign(xo) = +1 for xq > 0, and sign(xo) = — 1 if xq < 0. 
This turns out to ensure that H remains as close as possible to perfect orthogonality 
in the face of rounding errors. Because ||x||2 might be very large or very small, 
there is a risk of overflow or underflow in the computation of ||x||2. Thus, it 
is often better to compute y from jc/IWIoo- This works because scaling x does 
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not mathematically alter H (which may be confirmed as an exercise). Typically, 
m >> n (e.g., in the example of Fig. 4.1 we had m — N — 1000, while n — 3), 
so, since Hk € R mxm , we rarely can accumulate and store the elements of Hk for 
all k as too much memory is needed for such a task. Instead, it is much better to 
observe that (for example) if H e R mxm , A e R mxn then, as y e R m , we have 



HA 



T ~\ 

, yy 
y yl 



yTy 



y(A T y) T . 



(4.150) 



From (4.150) Ay e R", which has y'th element 

m — 1 



[A T y]j = Y] a k,jyk 



(4.151) 



k=Q 



for j =0, 1, 



1. If p = 2/y 1 y, then, from (4.150) and (4.151), we have 

~m — 1 



for i =0, 1,..., 

implements this is as follows: 

fl := 2/y T y; 

for; = to n - 1 do begin 

s := E^L"o a km 

S := ps; 

for / := to m - 1 do begin 



[HA]ij = a u - ^y; 



1, and j =0, 1, 



J2 a k,j 



yk 



.k=Q 



(4.152) 



1. A pseudocode program that 



end; 



end; 



This program is written to overwrite matrix A with matrix HA. This reduces 
computer system memory requirements. Recall (4.123), where we see that Q T f 
must be computed so that /" can be found. Knowledge of /" is essential to 
compute x via (4.126). As in the problem of computing HkA k ~ l , we do not wish 
to accumulate and save the factors Hk in 



Q 1 f = H„H„-i---Hif. 



(4.153) 



Instead, Q f would be computed using an algorithm similar to that suggested by 
(4.152). 

All the suggestions in the previous paragraph are needed in a practical imple- 
mentation of the Householder transformation matrix method for QR factorization. 
As noted in Ref. 5, the rounding error performance of the practical Householder 
QR factorization algorithm is quite good. It is stated [5] as well that the number of 
flops needed by the Householder method for finding x is greater than that needed by 
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Cholesky factorization. Somewhat simplistically, the Cholesky method is computa- 
tionally more efficient than the Householder method, but the Householder method 
is less susceptible to ill conditioning and to rounding errors than is the Cholesky 
method. More or less, there is therefore a tradeoff between speed and accuracy 
involved in selecting between these competing methods for solving the overde- 
termined least-squares problem. The Householder approach is also claimed [5] to 
require more memory than the Cholesky approach. 



4.7 ITERATIVE METHODS FOR LINEAR SYSTEMS 

Matrix A € R" x " is said to be sparse if most of its n 2 elements are zero-valued. 
Such matrices can arise in various applications, such as in the numerical solution 
of partial differential equations (PDEs). Sections 4.5 and 4.6 have presented such 
direct methods as the LU and QR decompositions (factorizations) of A in order to 
solve Ax — b (assuming that A is nonsingular). However, these procedures do not 
in themselves take advantage of any structure that may be possessed by A such 
as sparsity. Thus, they are not necessarily computationally efficient procedures. 
Therefore, in the present section, we consider iterative methods to determine x € R" 
in Ax = b. In this section, whenever we consider Ax — b, we will always assume 
that A -1 exists. Iterative methods work by creating a Cauchy sequence of vectors 
(x w ) that converges to x. 10 Iterative methods may be particularly advantageous 
when A is not only sparse, but is also large (i.e., large n). This is because direct 
methods often require the considerable movement of data around the computing 
machine memory system, and this can slow the computation down substantially. But 
a properly conceived and implemented iterative method can alleviate this problem. 

Our presentation of iterative methods here is based largely on the work of 
Quarteroni et al. [8, Chapter 4]. We use much of the same notation as that in 
Ref. 8. But it is a condensed presentation as this section is intended only to convey 
the main ideas about iterative linear system solvers. 

In Section 4.4 matrix and vector norms were considered in order to characterize 
the sizes of errors in the numerical estimate of x in Ax = b due to perturbations 
of A, and b. We will need to consider such norms here. As noted above, our goal 
here is to derive a methodology to generate vector sequence (x*^) 11 such that 

lim x (k) =x, (4.154) 

k—>oo 

where i = [io ii • • • x n -\\ T e R" satisfies Ax — b and x^ = [xq x\ ■ ■ ■ 
x^_j] r € R". The basic idea is to find an operator T such that x^ k+v> — Tx^ k \— 
T(x^)), for fc = 0,1,2,.... Because (x^) is designed to be Cauchy (recall 

As such, we will be revisiting ideas first seen in Section 3.2. 



Note that the "(k)" in x^' does not denote the raising of i to a power or the taking of the kxh 
derivative, but rather is part of the n 
the k\h power of A, but A^ is not. 



derivative, but rather is part of the name of the vector. Similar notation applies to matrices. So, A is 
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Section 3.2) for any e > 0, there will be an m e Z + such that ||x^ m ^ — x\\ < e 
[recall that d(x^ \ x) — \\x w — x\\]. The operator T is defined according to 

x (k+1) = Bx (k) + f, (4.155) 

where x^ e R" is the starting value (initial guess about the solution x), B e R" x " 
is called the iteration matrix, and / e R" is derived from A and b in Ax — b. Since 
we want (4.154) to hold, from (4.155) we seek B and / such that x — Bx + f, or 
A~ l b — BA~ l b + f (using Ax — b, implying x = A~ l b), so 

f = (I-B)A~ l b. (4.156) 

The error vector at step k is defined to be 

e (*) =,(*)_ X| (4.157) 

and naturally we want lim^oo e^ — 0. Convergence would be in some suitably 
selected norm. 

As matters now stand, there is no guarantee that (4.154) will hold. We achieve 
convergence only by the proper selection of B, and for matrices A possessing 
suitable properties (considered below). Before we can consider these matters we 
require certain basic results involving matrix norms. 

Definition 4.3: Spectral Radius Let s(A) denote the set of eigenvalues of 
matrix A e R" x ". The spectral radius of A is 

p(A) = max |A,|, 

X€s(A) 

An important property possessed by p(A) is as follows. 

Property 4.1 If A € R" x " with e > 0, then there is a norm denoted || • || e 
(i.e., a norm perhaps dependent on e) satisfying the consistency condition (4.36c), 
and such that 

\\A\\ € < P (A) + e. 

Proof See Isaacson and Keller [9]. 

This is just a formal way of saying that there is always a matrix norm that is 
arbitrarily close to the spectral radius of A 

p(A) = inf||A|| (4.158) 

with the infimum (defined in Section 1.3) taken over all possible norms that satisfy 
(4.36c). We say that the sequence of matrices (A^) [with A^ € R" xn ] converges 
to A € R" x " iff 

lim \\A (k) -A|| =0. (4.159) 
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The norm in (4.159) is arbitrary because of norm equivalence (recall discussion on 
this idea in Section 4.4). 

Theorem 4.7: Let A e R" x "; then 

lim A k = •»• p(A) < 1. (4.160) 

As well, the matrix geometric series YI^LqA converges iff p(A) < 1. In this 

instance 

oo 

J2A k = (I - A)~\ (4.161) 

k=Q 

So, if p(A) < 1, then matrix I — A is invertible, and also 

< ||(7 - A)" 1 !! < - \— , (4.162) 



1 + I|A||-" V "-1-IIAH 

where || • || here is an induced matrix norm (i.e., (4.36b) holds) such that ||A|| < 1. 

Proof We begin by showing (4.160) holds. Let p(A) < 1 so there must be an 
e > such that p(A) < 1 — e, and from Property 4.1 there is a consistent matrix 
norm 1 1 • 1 1 such that 

\\A\\ < p(A) + € < 1. 

Because [recall (4.40)] of ||A^|| < \\A\\ k < 1, and the definition of convergence, 
as k -* oo, we have A k -» € R" x ". Conversely, assume that lim^^oo A* = 0, 
and let X be any eigenvalue of A. For eigenvector x (^ 0) of A associated with 
eigenvalue X, we have A k x — X k x, and so \m\k^oo^ k — 0. Thus, |A| < 1, and 
hence p(A) < 1. Now consider (4.161). If X is an eigenvalue of A, then 1 — X is 
an eigenvalue of / — A. We observe that 

(/ - A)(I + A + A 2 + • • • + A" -1 + A") = / - A" +1 . (4.163) 

Since p(A) < \, I — A has an inverse, and letting n — > oo in (4.163) yields 



(/-A)^A* = / 



fc=0 

so that (4.161) holds. 

Now, because matrix norm || • || satisfies (4.36b), we must have ||/|| = 1. Thus 

1 = ||/|| < 11/ -A|| ||(7- A)" 1 !! < (l + ||A||)||(7- A)" 1 !!, 



TLFeBOOK 



ITERATIVE METHODS FOR LINEAR SYSTEMS 179 

which gives the first inequality in (4.162). Since / = (7 — A) + A, we have 



so that 



(I - A)' 1 = / + A(I - A)~ l 



|| ( /_ A) -i||< l + ||A|| ||(7- A)" 1 !]. 



Condition ||A|| < 1 implies that this yields the second inequality in (4.162). 

We mention that in Theorem 4.7 an induced matrix norm exists to give ||A|| < 
1 because of Property 4.1 (recall that (A®) is convergent, giving p(A) < 1). 
Theorem 4.7 now leads us to the following theorem. 

Theorem 4.8: Suppose that / e R" satisfies (4.156); then (x w ) converges to 
x satisfying Ax — b for any x^ iff p(B) < 1. 

Proof From (4.155)-(4.157), we have 

e (k+D = x (k+i) _ x = Bx (k) + f _ x = Bx (k) + (I _ B)A -i b _ x 

= Be {k) + Bx + (I - B)A~ l b - x 
= Be (k) + Bx + x- Bx -x 
= Be {k \ 



Immediately, we see that 



for k e Z + . From Theorem 4.7 



e » = BV 0) (4.164) 



lim B k e (0) = 

for all e (0) € R" iff p{B) < 1. 

On the other hand, suppose p(B) > 1; then there is at least one eigenvalue X of 
B such that |A.| > 1. Let e^ be the eigenvector associated with X, so Be^ — \e®\ 
implying that e^ — X k e < -°\ But this implies that e^' -/> as k — >■ oo since |A.| > 1. 

This theorem gives a general condition on B so that iterative procedure (4.155) 
converges. Theorem 4.9 (below) will say more. However, our problem now is to 
find B. From (4.158), and Theorem 4.7 a sufficient condition for convergence is 
that HZ? 1 1 < 1, for any matrix norm. 

A general approach to constructing iterative methods is to use the additive 
splitting of the matrix A according to 

A = P-N, (4.165) 
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where P, N e R nxn are suitable matrices, and P~ l exists. Matrix P is sometimes 
called a preconditioning matrix, or preconditioner (for reasons we will not consider 
here, but that are explained in Ref. 8). To be specific, we rewrite (4.155) as 



Ak+l) 



= P~ l Nx 



(k) 



P~ l b, 



that is, for k e Z H 



Px 



(*+i) 



Nx 



(k) 



(4.166) 



so that / = P~ l b, and B — P^N. Alternatively 



Ak+l) 




(4.167) 



where r^ is the residual vector at step k. From (4.167) we see that to obtain 
x (k+i) re q U j res us [ solve a linear system of equations involving P. Clearly, for 
this approach to be worth the trouble, P must be nonsingular, and be easy to invert 
as well in order to save on computations. 

We will now make the additional assumption that the main diagonal elements of 
A are nonzero (i.e., a,-,; ^ for all i e Z„). All the iterative methods we consider in 
this section will assume this. In this case we may express Ax = b in the equivalent 
form 



1 
an 






(4.168) 



for i — 0, 1 , . . . , n — 1 . 

The expression (4.168) immediately leads to, for any initial guess x®\ 
Jacobi method, which is defined by the iterations 



the 



,(*+!) 



n-\ 



J2 ai iA 



{10 



.7=0 



(4.169) 



for i =0, 1, 
splitting 



n — 1. It is easy to show that this algorithm implements the 

P = D,N = D- A = L + U, (4.170) 



where D — diag(ao,o, «i,i, ■ ■ ■ , «n-i,«-i) (i-e-, diagonal matrix that is the main 
diagonal elements of A), L is the lower triangular matrix such that Z, ; = — a,y 
if ( > j, and lu = if i < j, and [/ is the upper triangular matrix such that 
Uij — —au if j > i, and m (/ - = if j < i. Here the iteration matrix B is given by 



B = Bj = P~ [ N = D"'(L + [/) 



D~ l A. 



(4.171) 
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The Jacobi method generalizes according to 



(*+i) _ _^_ 

i 

an 



n-\ 



bi - ^ aijx ( j 



(k) 



j=0 



+ (1 - w)x\ 



(/<) 



(4.172) 



where i — 0, 1, . . . , n — 1, and a> is the relaxation parameter. Relaxation parame- 
ters are introduced into iterative procedures in order to control convergence rates. 
The algorithm (4.172) is called the Jacobi overrelaxation (JOR) method. In this 
algorithm the iteration matrix B takes on the form 



B = Bj{co) = wBj + (1 - co)I, 
and (4.172) can be expressed in the form (4.167) according to 

coD-'r^. 



^ +1 > = *<*> 



(4.173) 



(4.174) 



The JOR method satisfies (4.156) provided that co ^ 0. The method is easily seen 
to reduce to the Jacobi method when w = 1. 

An alternative to the Jacobi method is the Gauss-Seidel method. This is defined 
as 

1 



xf +1) = 



i—\ n—\ 

bi - ^ ajjx) + - ^2 a 'J x j 
,/=o y=i'+i 



(k) 



where i = 0, 1, . . . , n — 1. In matrix form (4.175) can be expressed as 



Dx^^b + Lx^ + Ux^, 



(4.175) 



(4.176) 



where D, L, and U are the same matrices as those associated with the Jacobi 
method. In the Gauss-Seidel method we implement the splitting 



D-L, N = U 



(4.177) 



with the iteration matrix 



B = B GS = {D-L)- l U. 



(4.178) 



As there is an overrelaxation method for the Jacobi approach, the same idea applies 
for the Gauss-Seidel case. The Gauss-Seidel successive overrelaxation (SOR) 
method is defined to be 



.(*+!) 



w 
an 



Y^a, 



. r (*+i) 
ij x j 



j=i+i 



.(*) 



+ (1 - co)x\ 



(k) 



(4.179) 
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again for i = 0, 1, ...,« — 1. In matrix form this procedure can be expressed as 

Dx (k+1) = co[b + Lx (k+l) + Ux (k) ] + (1 - co)Dx (k) 

or 

[/ - coD~ l L]x {k+l) = &>D _1 fr + [(1 - w)I + wD~ l U]x (k) , (4.180) 

for which the iteration matrix is now 

B = B GS (a}) = [I -o)D- l L]- l [(l - co)I + wD~ l U]. (4.181) 

We see from (4.180) (on multiplying both sides by D) that 

[D - aL]x (k+l) =wb+ [(1 - co)D + oU]x (k) , 
so from the fact that A = D — (L + U) [recall (4.170)], this may be rearranged as 



x (k+\) = x (k) 



1 

-D-L 

a> 



r (k) (4.182) 



(r (k) = b- Ax (k) ), which is the form (4.167). Condition (4.156) holds if w ^ 0. 
The case a> — 1 corresponds to the Gauss-Seidel method in (4.175). If a> e (0, 1), 
the technique is often called an underrelaxation method, while for a> e (1, oo) it 
is an overrelaxation method. 

We will now summarize, largely without proof, results concerning the conver- 
gence of (x w ) to x for sequences generated by the previous iterative algorithms. 
We observe that every iteration in any of the proposed methods needs (in the worst 
case, assuming that A is not sparse) 0(n 2 ) arithmetic operations. The total number 
of iterations is m, and is needed to achieve desired accuracy \\x^ mS) — x\\ < e, and 
so in turn the total number of arithmetic operations needed is 0(mn 2 ). Gaussian 
elimination needs 0(« 3 ) operations to solve Ax — b, so the iterative methods are 
worthwhile computationally only if m is sufficiently small. If m is about the same 
size as n, then little advantage can be expected from iterative methods. On the 
other hand, if A is sparse, perhaps possessing only 0(n) nonzero elements, then 
the iterative methods require only 0(mn) operations to achieve ||x^ m ^ — x\\ < e. 
We need to give conditions on A so that x^' — >• x, and also to say something about 
the number of iterations needed to achieve convergence to desired accuracy. 

Let us begin with the following definition. 

Definition 4.4: Diagonal Dominance Matrix A € R nx " is diagonally domi- 
nant if 

n-\ 



J] lay | (4.183) 



.7=0 



for i — 0, 1 , . . . , n — 1 . 
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We mention here that Definition 4.4 is a bit different from Definition 6.2 (Chapter 6), 
where diagonal dominance concepts appear in the context of spline interpolation 
problems. It can be shown that if A in Ax = b is diagonally dominant according to 
Definition 4.4, then the Jacobi and Gauss-Seidel methods both converge. Proof for 
the Jacobi method appears in Theorem 4.2 of Ref. 8, while the Gauss-Seidel case is 
proved by Axelsson [10]. 

If A — A T , and A > both the Jacobi and Gauss-Seidel methods will converge. 
A proof for the Gauss-Seidel case appears in Golub and Van Loan [5, Theo- 
rem 10.1.2]. The Jacobi case is considered in Ref. 8. Convergence results exist for 
the overrelaxation methods JOR and SOR. For example, if A — A T with A > 0, 
the SOR method is convergent iff < w < 2 [8]. Naturally, we wish to select a> so 
that convergence occurs as rapidly as possible (i.e., m in ||jc^ — x|| < e is min- 
imal). However, the problem of selecting the optimal value for a> is well beyond 
the scope of this book. 

We recall that our iterative procedures have the general form in (4.155), where 
it is intended that x — Bx + f. We may regard y — Tx — Bx + f as a mapping 
r|R" — > R". On linear vector space R" we may define the metric 

d(x, y) — max \xj - yj\ (4.184) 

(recall the properties of metrics from Chapter 1). Space (R", d) is a complete metric 
space [11, p. 308]. From Kreyszig [11] we have the following theorem. 

Theorem 4.9: If the linear system x = Bx + f is such that 

M-l 

£iM<! 

.7=0 

for i = 0, 1 , . . . , n — 1 then solution x is unique. The solution can be obtained as 
the limit of the vector sequence (x™) for k — 0, 1,2,... (x^ is arbitrary), where 

x (k+i) = Bx (k) + ff 
and where for a — max,-^ J]"Z \bij\, we have the error bounds 

in 

d(x (m \x) < — ^C-'^W) < -?—d(x (0 \x m ). (4.185) 

1 — a 1 — a 

Proof We will give only an outline proof. This theorem is really just a special 
instance of the contraction theorem, which appears and is proved in Chapter 7 (see 
Theorem 7.3 and Corollary 7.1). 

The essence of the proof is to consider the fact that 



d(Tx, Ty) — max 
ieZ„ 



n-l 
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n-1 

< max \xj — y ,■ I max V^ \ba I 

;eZ„ !6Z„ ^— ' 

.,=0 

n-1 

= d(x,y)max^|%|, 

,£Z " .7=0 



so d(Tx, Ty) < ad(x, y), if we define 



n-1 



[recall (4.41d)]. 



In this theorem we see that if a < 1, then jp" — > jc. In this case d(Tx, Ty) < 
cf(x, 3O for all ;t, y e R". Such a mapping r is called a contraction mapping (or 
contractive mapping). We see that contraction mappings have the effect of moving 
points in a space closer together. The error bounds stated in (4.185) give us an 
idea about the number of iterations m needed to achieve ||x^ m ^ — x\\ < e (e > 
0). We emphasize that condition a < 1 is sufficient for convergence, so (x^) 
may converge to x even if this condition is violated. It is also noteworthy that 
convergence will be fast if a is small, that is, if ||fi||oo i s small. The result in 
Theorem 4.8 certainly suggests convergence ought to be fast if p(B) is small. 

Example 4.8 We shall consider the application of SOR to the problem of 
solving Ax — b, where 



4 10 
14 10 
14 1 
14 



b = 



1 
2 
3 

4 



We shall assume that jt (0) = [0000] T . Note that SOR is not the best way to solve 
this problem. A better approach is to be found in Section 6.5 (Chapter 6). This 
example is for illustration only. However, it is easy to confirm that 

x = [0.1627 0.3493 0.4402 0.8900] r . 



Recall that the SOR iterations are specified by (4.179). However, we have not 
discussed how to terminate the iterative process. A popular choice is to recall that 



,(*) 



b — Ax^' [see (4.167)], and to stop the iterations when for k 



Am)\ 



-(0)1 



< T 



(4.186) 
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Number of iterations needed by SOR 



150 



100 




Figure 4.4 Plot of the number of iterations m needed by SOR as a function of co for the 



parameters of Example 4.8 in order to satisfy the stopping condition ||r' m - ) | loo/Ik 



r(0| 



< T. 



for some r > (a small value). For our present purposes 1 1 • 1 1 shall be the norm 
in (4.29c), which is compatible with the needs of Theorem 4.9. We shall choose 
t = 0.001. 

We observe that A is diagonally dominant, so convergence is certainly expected 
for co — 1. In fact, A > so convergence of the SOR method can be expected for 
all co e (0, 2). Figure 4.4 plots the m that achieves (4.186) versus co, and we see 
that there is an optimal choice for co that is somewhat larger than co — 1 . In this 
case though the optimal choice does not lead to much of an improvement over 
choice o)=l. 

For our problem [recalling (4.170)], we have 





' 4 C 


" 

















" 




D = 


4 
C 




4 


,L = 




-1 





-1 










. 




_ ° c 


4 









- 


-1 


_ 






" - 


-1 


" 












U = 







-1 






-1 

_ 












(4.178) 


" 0.0000 


-0.2500 




0.0000 




0.0000 


Bgs — 


0.0000 
0.0000 


0.0625 
-0.0156 




-0.2500 

0.0625 




0.0000 
-0.2500 






0.0000 


0. 


00 


39 




-0.0156 




0.06 


25 



We therefore find that ||BgsIIoo = 0.3281. It is possible to show (preferably using 
MATLAB or some other software tool that is good with eigenproblems) that 
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P(Bgs) — 0.1636. Given (4.185) in Theorem 4.9 we therefore expect fast con- 
vergence for our problem since a is fairly small. In fact 

ll* (w) -*||oo< ; "^""ji IKP-^r^Hoo (4.187) 

1 - WdgsWoo 

[using x^ — 0, x^ — (D — L)~ l b]. For the stopping criterion of (4.186) we 
obtained (recalling that w = 1, and r=0.001)m = 5 with 

x (5) = [0.1630 0.3490 0.4403 0.8899] r 

so that 

||jc c5) — jcMoo = 3.6455 x 10 -4 . 

The right-hand side of (4.187) evaluates to 

i/,,t;s!i '"' ; <\(D -L)- l b\\oo = 4.7523 x 10" 3 . 



l-\\B G s\ 
Thus, (4.187) certainly holds true. 

4.8 FINAL REMARKS 

We have seen that inaccurate solutions to linear systems of equations can arise 
when the linear system is ill-conditioned. Condition numbers warn us if this is 
a potential problem. However, even if a problem is well-conditioned, an inac- 
curate solution may arise if the algorithm applied to solve it is unstable. In the 
case of problems arising out of algorithm instability, we naturally replace the 
unstable algorithm with a stable one (e.g., Gaussian elimination may need to 
be replaced by Gaussian elimination with partial pivoting). In the case of an 
ill-conditioned problem, we may try to improve the accuracy of the solution by 
either 

1 . Using an algorithm that does not worsen the conditioning of the underlying 
problem (e.g., choosing QR factorization in preference to Cholesky factor- 
ization) 

2. Reformulating the problem so that it is better conditioned 

We have not considered the second alternative in this chapter. This will be done 
in Chapter 5. 

APPENDIX 4.A HILBERT MATRIX INVERSES 

Consider the following MATLAB code: 

R = hilb(10); 
inv(R) 
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ans 



1 .Oe+12 * 



Columns 1 through 7 






.0000 


-0 


,0000 


0, 


,0000 


-0, 


,0000 





,0000 


-0 


.0000 





.0000 





.0000 





.0000 


-0 


,0000 





,0000 


-0 


,0002 





.0005 


-0 


.0008 





.0000 


-0 


,0000 


0. 


,0001 


-0, 


,0010 





,0043 


-0 


.0112 





.0178 





.0000 





.0000 


-0 


,0010 





,0082 


-0 


,0379 





.1010 


-0 


.1616 





.0000 


-0 


,0002 


0, 


,0043 


-0, 


,0379 





,1767 


-0 


.4772 





.7712 





.0000 





.0005 


-0 


,0112 





,1010 


-0 


,4772 


1 


.3014 


-2 


.1208 





.0000 


-0 


,0008 


0. 


,0178 


-0, 


,1616 





,7712 


-2 


.1208 


3 


.4803 





.0000 





.0008 


-0 


,0166 





,1529 


-0 


,7358 


2 


.0376 


-3 


.3636 





.0000 


-0 


,0004 


0, 


,0085 


-0, 


,0788 





,3820 


-1 


.0643 


1 


.7659 





.0000 





.0001 


-0 


,0018 





,0171 


-0 


,0832 





.2330 


-0 


.3883 



Columns 8 through 10 



-0.0000 





.0000 


-0 


,0000 


0.0008 


-0 


.0004 


0. 


,0001 


-0.0166 





.0085 


-0 


,0018 


0.1529 


-0 


.0788 


0, 


,0171 


-0.7358 





.3820 


-0 


,0832 


2.0376 


-1 


.0643 


0, 


,2330 


-3.3636 


1 


.7659 


-0 


,3883 


3.2675 


-1 


.7231 


0, 


,3804 


-1 .7231 





.9122 


-0 


,2021 


0.3804 


-0 


.2021 


0, 


,0449 


R*inv(R) 











Columns 1 through 7 



1 


,0000 





,0000 





.0000 


-0 


.0000 





,0001 





.0001 


-0 


.0001 





,0000 


1 


,0000 





.0000 


-0 


.0000 





,0001 





.0001 


-0 


.0002 





,0000 





,0000 


1 


.0000 


-0 


.0000 





,0001 





.0000 


-0 


.0001 





,0000 





,0000 





.0000 


1 


.0000 





,0000 





.0000 


-0 


.0000 





,0000 





,0000 





.0000 


-0 


.0000 


1 


,0000 


-0 


.0000 


-0 


.0000 





,0000 





,0000 





.0000 


-0 


.0000 





,0000 


1 


.0000 


-0 


.0000 





,0000 





,0000 





.0000 


-0 


.0000 





,0000 





.0000 





.9999 





,0000 





,0000 





.0000 


-0 


.0000 





,0000 





.0001 


-0 


.0000 





,0000 





,0000 





.0000 


-0 


.0000 





,0000 





.0000 


-0 


.0001 





,0000 





,0000 





.0000 


-0 


.0000 





,0000 





.0000 


-0 


.0000 



Columns 8 through 10 



-0.0000 
0.0001 
0.0000 

-0.0000 



-0.0001 
-0.0001 
-0.0001 
-0.0000 



0.0000 
0.0000 
0.0000 
0.0000 
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0.0000 


-0 


,0000 





.0000 


-0.0000 


-0 


,0000 





.0000 


0.0000 


-0 


,0000 





.0000 


1 .0000 


-0 


.0000 





,0000 


0.0000 


1 


.0000 





,0000 


-0.0000 


-0 


.0000 


1 


,0000 


R = hilb(11) 


) 








inv(R) 











1 .Oe+14 * 



Columns 1 through 7 






,0000 


-0 


,0000 





.0000 


-0 


,0000 





.0000 


-0 


,0000 





,0000 





.0000 





.0000 


-0 


.0000 





.0000 


-0 


,0000 





,0000 


-0 


.0000 





,0000 


-0 


,0000 





.0000 


-0 


,0000 





,0002 


-0 


,0006 





,0012 





,0000 





.0000 


-0 


,0000 





.0003 


-0 


,0019 





,0064 


-0 


.0137 





,0000 


-0 


,0000 





,0002 


-0 


,0019 





,0110 


-0 


,0381 





,0817 





.0000 





,0000 


-0 


.0006 





.0064 


-0 


,0381 





,1329 


-0 


,2877 





,0000 


-0 


,0000 





.0012 


-0 


,0137 





,0817 


-0 


,2877 





,6270 





,0000 





,0001 


-0 


.0016 





.0183 


-0 


,1101 





,3902 


-0 


,8555 





,0000 


-0 


.0000 





.0013 


-0 


,0149 





,0905 


-0 


,3227 





.7111 





.0000 





.0000 


-0 


.0006 





.0068 


-0 


,0415 





,1487 


-0 


,3292 





,0000 


-0 


.0000 





.0001 


-0 


,0013 





,0081 


-0 


,0293 





,0651 



Columns 8 through 11 



-0 


.0000 





.0000 


-0, 


,0000 





.0000 





,0001 


-0 


,0000 


0, 


,0000 


-0 


,0000 


-0 


.0016 





,0013 


-0, 


,0006 





.0001 





,0183 


-0 


,0149 


0, 


,0068 


-0 


,0013 


-0 


.1101 





.0905 


-0, 


.0415 





.0081 





,3902 


-0 


.3227 


0, 


,1487 


-0 


,0293 


-0 


,8555 





,7111 


-0, 


,3292 





.0651 


1 


,1733 


-0 


,9796 


0, 


,4553 


-0 


,0903 


-0 


,9796 





.8212 


-0, 


,3830 





.0762 





,4553 


-0 


,3830 


0, 


,1792 


-0 


,0357 


-0 


,0903 





,0762 


-0. 


,0357 





.0071 


R*inv< 


(R) 















ans 



Columns 1 through 7 






,9997 


-0 


,0009 





,0022 





,0028 


-0 


,0164 





,0558 


-0 


,1229 





.0002 





,9992 





.0020 





.0023 


-0 


,0132 





.0454 


-0, 


,1029 





.0002 


-0 


,0007 


1 


.0018 





.0019 


-0 


,0112 





,0385 


-0. 


,0844 





.0002 


-0 


,0006 





.0016 


1 


.0017 


-0 


,0097 





,0331 


-0, 


,0736 





,0002 


-0 


,0006 





,0015 





.0015 





,9915 





,0285 


-0 


,0638 





.0002 


-0 


,0005 





.0014 





,0013 


-0 


,0076 


1 


.0258 


-0, 


,0581 
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-0 


.0002 


-0 


,0005 





,0013 





,0012 


-0 


,0070 


-0 


.0001 


-0 


,0005 





,0012 





,0011 


-0 


,0063 


-0 


.0001 


-0 


,0004 





,0011 





,0010 


-0 


,0059 


-0 


.0001 


-0 


,0004 





,0010 





,0009 


-0 


,0053 


-0 


.0001 


-0 


,0004 





,0010 





,0009 


-0 


,0052 



0.0234 0.9468 

0.0216 -0.0474 

0.0201 -0.0448 

0.0187 -0.0406 

0.0179 -0.0395 



Columns 8 through 11 






,1665 


-0, 


,1405 





.0652 


-0. 


,0091 





,1351 


-0, 


,1165 





.0530 


-0, 


,0071 





,1125 


-0, 


.0973 





.0452 


-0. 


,0058 





,0964 


-0, 


.0844 





.0385 


-0. 


,0047 





,0858 


-0, 


,0739 





.0341 


-0, 


.0041 





,0745 


-0, 


,0661 





.0300 


-0. 


,0037 





,0696 


-0, 


.0592 





.0280 


-0, 


,0033 


1 


,0635 


-0, 


,0547 





.0251 


-0. 


,0029 





,0581 


0, 


,9495 





.0235 


-0 


,0028 





,0536 


-0, 


,0458 


1 


.0213 


-0, 


,0024 





,0527 


-0, 


.0445 





.0207 





,9976 



R = hilb(12); 

inv(R) 

Warning: Matrix is close to singular or badly scaled. 

Results may be inaccurate. RC0ND = 2.632091e-17. 



1 .Oe+15 * 



Columns 1 through 7 






,0000 


-0 


,0000 


0. 


.0000 


-0, 


,0000 


0, 


.0000 


-0 


.0000 





.0000 





.0000 





.0000 


-0 


,0000 





,0000 


-0, 


.0000 





.0000 


-0 


.0000 





,0000 


-0 


,0000 





,0000 


-0. 


,0000 


0, 


,0001 


-0 


.0002 





.0006 





.0000 





.0000 


-0 


,0000 





,0001 


-0, 


.0008 





.0032 


-0 


.0086 





,0000 


-0 


,0000 


0. 


,0001 


-0. 


,0008 


0, 


,0054 


-0 


.0229 





.0624 





.0000 





.0000 


-0 


,0002 





,0032 


-0, 


.0229 





.0990 


-0 


.2720 





,0000 


-0 


,0000 


0. 


,0006 


-0. 


,0086 


0, 


.0624 


-0 


.2720 





.7528 





.0000 





.0000 


-0 


,0011 





,0151 


-0, 


,1107 





.4863 


-1 


.3545 





,0000 


-0 


,0000 


0. 


,0013 


-0. 


,0173 


0, 


.1276 


-0 


.5640 


1 


.5794 





,0000 





,0000 


-0 


,0009 





,0124 


-0, 


.0920 





.4090 


-1 


.1511 





,0000 


-0 


,0000 


0. 


,0004 


-0, 


,0050 


0, 


.0377 


-0 


.1686 





.4765 





.0000 





.0000 


-0 


,0001 





,0009 


-0, 


.0067 





.0301 


-0 


.0855 



Columns 8 through 12 






.0000 





.0000 


-0 


,0000 





,0000 


-0 


,0000 





,0000 


-0 


,0000 


0, 


,0000 


-0, 


,0000 





,0000 





.0011 





.0013 


-0 


,0009 





,0004 


-0 


,0001 





,0151 


-0 


,0173 


0. 


,0124 


-0. 


,0050 





,0009 





.1107 





.1275 


-0 


,0920 





,0377 


-0 


,0067 





,4863 


-0 


,5639 


0, 


,4090 


-0, 


,1686 





,0301 


1 


.3544 


1 


.5793 


-1 


.1510 





,4765 


-0 


,0855 


2 


,4505 


-2 


,8712 


2. 


,1015 


-0. 


,8732 





,1572 



TLFeBOOK 



190 



LINEAR SYSTEMS OF EQUATIONS 



-2.8713 3.3786 -2.4821 

2.1016 -2.4822 1.8297 

-0.8732 1.0348 -0.7651 

0.1572 -0.1869 0.1386 



1.0348 -0.1869 

-0.7651 0.1385 

0.3208 -0.0582 

-0.0582 0.0106 



R*inv(R) 

Warning: Matrix is close to singular or badly scaled. 

Results may be inaccurate. RC0ND = 2.632091 e-17. 



Columns 1 through 7 



1 


,0126 


-0 


,0066 


-0 


,0401 





,0075 





,1532 


-0 


,8140 


2 


.2383 





,0113 





.9943 


-0 


,0361 





,0100 





,1162 


-0 


,6265 


1 


.7168 





,0103 


-0 


,0050 





,9673 





,0106 





,0952 


-0 


,5300 


1 


.4834 





,0094 


-0 


,0045 


-0 


,0299 


1 


,0104 





,0797 


-0 


,4573 


1 


.2725 





,0087 


-0 


,0041 


-0 


,0275 





,0104 


1 


,0703 


-0 


,4038 


1 


.0986 





,0081 


-0 


,0037 


-0 


,0255 





,0102 





,0621 





,6416 





.9971 





,0075 


-0 


.0034 


-0 


,0237 





.0099 





,0554 


-0 


,3245 


1 


.9062 





,0071 


-0 


,0032 


-0 


,0222 





,0095 





,0495 


-0 


,2944 





.8232 





,0066 


-0 


,0030 


-0 


,0209 





.0093 





,0439 


-0 


,2686 





.7246 





,0063 


-0 


,0028 


-0 


,0197 





,0087 





,0424 


-0 


,2532 





.7002 





,0059 


-0 


,0026 


-0 


,0187 





,0087 





,0372 


-0 


,2307 





.6309 





,0057 


-0 


.0024 


-0 


,0177 





.0081 





,0358 


-0 


,2168 





.6064 



Columns 8 through 12 



-4 


.0762 


4 


.7656 


-3 


.5039 


1 


,4385 


-0 


,2183 


-3 


.1582 


3 


.7754 


-2 


.7520 


1 


,1123 


-0 


,1649 


-2 


.6250 


3 


.1055 


-2 


.3301 





.9219 


-0 


,1390 


-2 


.2676 


2 


.6602 


-1 


.9922 





,7905 


-0 


,1163 


-1 


.9863 


2 


.4023 


-1 


.7139 





,7104 


-0 


,0992 


-1 


.7969 


2 


.1094 


-1 


.5430 





,6289 


-0 


,0897 


-1 


.6133 


1 


.9258 


-1 


.4043 





,5581 


-0 


,0779 


-0 


.4658 


1 


.7598 


-1 


.2734 





,5146 


-0 


,0715 


-1 


.3047 


2 


.5762 


-1 


.1445 





,4629 


-0 


,0651 


-1 


.2793 


1 


.5098 


-0 


.1055 





,4424 


-0 


,0619 


-1 


.1387 


1 


.3438 


-0 


.9873 


1 


,3955 


-0 


,0529 


-1 


.1025 


1 


.2998 


-0 


.9395 





,3809 





,9474 


diary 


Off 



















The MATLAB rcond function (which gave the number RCOND above) needs 
some explanation. A useful reference on this is Hill [3, pp. 229-230]. It is based on 
a condition number estimator in the old FORTRAN codes known as "LINPACK". 
It is based on 1-norms. rcond(A) will give the reciprocal of the 1-norm condition 
number of A. If A is well-conditioned, then rcond(A) will be close to unity (i.e., 
close to one), and will be very tiny if A is ill-conditioned. The rule of thumb 
involved in interpreting an rcond output is "if rcond(A) « d x \Q~ k , where d is 
a digit from 1 to 9, then the elements of xcomp can usually be expected to have 
k fewer significant digits of accuracy than the elements of A" [3]. Here xcomp 
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is simply the computed solution to Ax — y; that is, in the notation of the present 
set of notes, x =xcomp. MATLAB does arithmetic with about 16 decimal digits 
[3, p. 228], so in the preceding example of a Hilbert matrix inversion problem for 
N — 12, since RCOND is about 10~ 17 , we have lost about 17 digits in computing 
R~ l . Of course, this loss is catastrophic for our problem. 



APPENDIX 4.B SVD AND LEAST SQUARES 

From Theorem 4.4, A — lfEV T , so this expands into the summation 

p-i 
A = Y,°iUivJ- (4.A.1) 

i=a 

But if r — rank (A), then (4.A.1) reduces to 

r-l 

A = ^2a iUi vJ. (4.A.2) 

i'=0 

In the following theorem p 2 LS — \\f l \\\ — \\Ax — f\\^ [see (4.127)], and x is the 
least-squares optimal solution to Ax — f . 

Theorem 4.B.1: Let A be represented as in (4.A.2) with A € R mx " and m > n. 
If / € R m then 

*-§(^r)"- <4A3) 

m — 1 



Proof For all x e R", using the invariance of the 2-norm to orthogonal trans- 
formations, and the fact that VV T — I n (n x n identity matrix) 

\\Ax - f\\\ = \\U T AV(V T x) - U T f\\ 2 2 = ||Ea - U T f\\ 2 2 , (4.A.5) 

where a — V T x, so as a — [ao ■ ■ -Q!„_i] r we have a ; - = vj x. Equation (4. A. 5) 
expands as 



\Ax - f\\\ = a 1 Y* 1 Ea - 2a 1 Y, 1 U 1 f + f 1 UU 1 f, (4.A.6) 
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which further expands as 

r—\ r—\ m—\ 



\ax - f\\\ = J2 a ? a ? - 22><W7 + £t«f /] 2 

i=0 i'=0 i=0 

r—\ m — \ 

= i> 2 «, 2 - ^wif + [ M < r /] 2 ] + i>r /i 2 

i— i=r 

r—\ m — \ 

= J2l°i<*i - "If? + I>f /l 2 - (4-A.7) 



(=0 i=r 

To minimize this we must have CT,a,- — uj f — 0, and so 

«i = -wf/ (4.A.8) 

for ( € Z r . As a = V r x, we have x — Va, so if we set a r — ■ ■ ■ — a n -\ — 0, then 
from Eq. (4. A. 8), we obtain 

r— 1 r— 1 y /. 

which is (4. A. 3). For this choice of jc from (4.A.7) 

m— 1 

ii^-/H2 = I>J7] 2 = pL> 

which is (4.A.4). 

Define A + = VE + £/ r (again A e R mx " with m > «), where 

E+ = diagCcr" 1 , . . . , a~\ , 0, . . . , 0) € R" xm . (4.A.9) 

We observe that 

r—\ j f 

A+/ = VY + U T f = V— 1>; = x. (4.A.10) 

i=0 ' 

We call A + the pseudoinverse of A. We have established that if rank (A) = n, 
then A T Ax — A T f, soi = (A^A) - ^ 7 /, which implies that in this case A + = 
(A T A)- l A T . If A € R" x " and A -1 exists, then A+ = A -1 . 

If A -1 exists (i.e., m — n and rank (A) = n), then we recall that K2(A) = 
II^I|2||A _1 ||2- If A € R mx " and m > n, then we extend this definition to 

K 2 (A) = ||A|| 2 ||A + || 2 . (4.A.11) 
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We have established that ||A|| 2 = o"o so since A + = VE + U T we have ||A + ||2 = 
l/cr r _i . Consequently 

K2 (A) = — (4.A.12) 

a r -\ 

which provides a somewhat better justification of (4.119), because if rank (A) = 
n then (4.A.12) is k 2 (A) = ao/<r n -i = or max (A)/(r min (A) [which is (4.119)]. 
From (4.A.11), K 2 (A r A) = ||A r A|| 2 ||(A r A)+|| 2 . With A = (/Sy r , and A r = 
V£ r {/ r we have 

a t a = vz T i:v T , 

and 

(A r A)+ = y(E r E)+y r . 

Thus, ||A r A|| 2 = cr 2 , and \\(A T A)+\\ 2 = ff^ (rank (A) = «). Thus 



iTl^ "0 _ r / 4 s n 2 



k 2 (A j A) = -^- = [k 2 (A)Y 



The condition number definition k p (A) in (4.60) was fully justified because of 
(4.58). An analogous justification exists for (4. A. 11), but is much more difficult to 
derive, and this is why we do not consider it in this book. 
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PROBLEMS 

4.1. Function f(x) e L 2 [0, 1] is to be approximated according to 

a\ 

f(x) « a x H ■ — 

X + c 

using least squares, where c e R is some fixed parameter. This involves 
solving the linear system 



I 1 - clog, (l + 1) 



c(c+l) 



«0 

5] 



Jo */(*) rfx 

' /M 



JO x+c aX 



= R 



where a is the vector from R that minimizes the energy V(a) [Eq. (4.8)]. 

(a) Suppose that f(x) — x + -i-. Find a for c = 1. For this special case 
it is possible to know the answer in advance without solving the linear 
system above. However, this problem requires you to solve the system. 
[Hint: It helps to recall that 





a b 
c d 




1 


d 
—c 


-b 




ad — be 


a 


(b) Derive R [which is a special case of (4.9)]. 




4.2. Suppose that 


A 


= 


1 2 " 
_ -1 -5 







Find IIAHoo, ||A||i, and ||A|| 2 
4.3. Suppose that 

A = 



1 

2 6 



€ R 2x2 ,e > 0, 



and that A ' exists. Find Koo(A) if e is small. 
4.4. Suppose that 



A = 



1 1 

6 



eR 



2x2 



and assume e > (so that A ' always exists). Find K2(A) — ||A||2||A '||2. 
What happens to condition number /C2(A) if e -> 0? 

4.5. Let A(e), B(e) € R nxn . For example, A(e) = [fl,/(e)] so element fl !; (e) of 
A(e) depends on the parameter e € R. 
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(a) Prove that 

d dB(e) dA(e) 

— [A(e)B(e)] = A(e)— -^ + — -^fl(e), 
de de de 

where dA(e)/dt — [datj(e)/de], and dB(e)/dt = [dbij(e)/de]. 

(b) Prove that 

dA(e)~ 



— A- i (e) = -A- 1 (e) 
de 



de 



A~\e). 



[Hint: Consider £[A(e) A" 1 (e)] = £l = 0, and use (a).] 

4.6. This problem is an alternative derivation of k(A). Suppose that e e R, A, F e 
R" x " , and je (e) , y , f e R" . Consider the perturbed linear system of equations 

(A + eF)x(e) = y + ef, (4.P.1) 

where Ax = y, so x(0) = x is the correct solution. Clearly, eF models the 
errors in A, while ef models the errors in y. From (4.P.1), we obtain 

x(e) = [A + eF]- l (y + ef). (4.P.2) 

The Taylor series expansion for x(e) about e = is 

dx(0) -, 

X ( 6 )=x(0) + — y —L e + 0(e 2 ), (4.P.3) 

de 

where 0(e 2 ) denotes terms in the expansion containing e k for k > 2. 

Use (4.P.3), results from the previous problem, and basic matrix-vector norm 
properties (Section 4.4) to derive the bound in 

||jc(e) — jcII , f||/|| IIFII 

-^—^ < ellAHIIA -1 !! ' ' ' 



=K(A) 



\\y\\ \\A\\ 



[Comment: The relative error in A is p a — e||F||/||A||, and the relative error 
in y is p y — e\\f\\/\\y\\, so more concisely \\x(e) — x\\/\\x\\ < K(A)(p A + 

Py)-1 

4.7. This problem follows the example relating to Eq. (4.95). An analog signal 
/(f) is modeled according to 

p-i 
f(t) = Y^cijt j +r)(t), 

.7=0 

where a ; - € R for all j e Z p , and n(t) is some random noise term. We only 
possess samples of the signal; that is, we only have the finite-length sequence 
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(/„) defined by f n — f(nT s ) for n e Zyv, where T s > is the sampling 
period of the data acquisition system that gave us the samples. Our estimate 
of /„ is therefore 

f n = J2 a J T s n '- 

;'=o 



With a — [«o oi 



flp-2 a p -i] T e R p , find 

JV-l 
V(fl)= J^lfn-fnf 



«=0 

in the form of Eq. (4.102). This implies that you must specify p, g, A, and P. 

4.8. Sampled data /„ (n = 0, 1, . . . , N — 1) is modeled according to 

f n — a + bn + C sm(0n + (f>). 

Recall sin(A + B) = sin A cos B + cos A sin B. Also recall that 

AT— l 
Vm = £[/n - /«] 2 = / r /~ / r A(0)[A r (0)A(0)]- 1 A r (0)/, 

n=0 

where / = [/o/i • • • /n-iV € R'*'. Give a detailed expression for A(0). 

4.9. Prove that the product of two lower triangular matrices is a lower triangular 
matrix. 

4.10. Prove that the product of two upper triangular matrices is an upper triangular 
matrix. 

4.11. Find the LU decomposition of the matrix 



2 


-1 





1 


2 


-1 





-1 


2 



using Gauss transformations as recommended in Section 4.5. Use A — LU to 
rewrite the factorization ofAasA = LZ)L r , where L is unit lower triangular, 
and D is a diagonal matrix. Is A > 0? Why? 

4.12. Use Gaussian elimination to LU factorize the matrix 



A = 



l 

L 2 



4 -1 

-1 4 

-1 



l -\ 

2 
-1 

4 



Is A > 0? Why" 
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4.13. (a) Consider A e R" x ", and A — A T . Suppose that the leading principal 

submatrices of A all have positive determinants. Prove that A > 0. 

(b) Is 

5-3 

-3 5 1 > ()■' 
15 

Why? 

4.14. The vector space R" is an inner product space with inner product (x, y) — 
x T y for all x, y e R". Suppose that A e R" x ", A = A T , and also that A > 0. 
Prove that (x, y) — x T Ay is also an inner product on the vector space R". 

4.15. In the quadratic form 



V(x) = ff 



2x T A T f + x T A T Ax, 



we assume x e R", and A T A > 0. Prove that 



V(x) = f'f-f 1 A[A 1 A]- 1 A 1 f, 
where x is the vector that minimizes V(x). 



4.16. Suppose that A e R NxM with M < N, and rank (A) = M. Suppose also that 
(x, y) = x T y. Consider P = A[A T A] -1 A T , Pj_ — I - P (I is the N x N 
identity matrix). Prove that for all x e R N we have (Px, P±x) — 0. (Com- 
ment: Matrices P and P± are examples of orthogonal projection operators.) 

4.17. (a) Write a MATLAB function for forward substitution (solving Lz — y). 

Write a MATLAB function for backward substitution (solving Ux — z). 
Test your functions out on the following matrices and vectors: 



v 



[1 



1 


o - 




- 1 


2 


3 4 


1 1 

| 1 






, u = 






3 



5 5 

7 1 

3 3 


-f 


1 _ 




. 





f 


1 -1 - 


if 


.* = [! 


2 


7 

3 


~2] T - 



(b) Write a MATLAB function to implement the LU decomposition algo- 
rithm based on Gauss transformations considered in Section 4.5. Test 
your function out on the following A matrices: 



A = 



12 3 4 

-112 1 

2 13 

11 



A = 



2-1 

-1 2 -1 

0-1 2 
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100 n 




100 V 



100 n 



20 n 



50 Q 



1 A 

MS) 



8Q 




200 V 



10Q 



1000 a 



100 n 



Figure 4.P.1 The DC electric circuit of Problem 4.18. 

4.18. Consider the DC electric circuit in Fig. 4.P. 1 . 

Write the node equations for the node voltages V\, V2, ■ ■ ■ , V(, as shown. 
These may be loaded into a vector 

v = [V^VsV^sVef 

such that the node equations have the form 

Gv = y. (4.P.4) 

Use the Gaussian elimination, forward substitution, and backward substitu- 
tion MATLAB functions from the previous problem to solve the linear system 
(4.P.4). 

4.19. Let /„ € R" x " be the order n identity matrix, and define 



a 


b 


• 


• 





b 


a 


b ■ 


• 








b 


a 


• 











• 


a 


b 








• 


■ b 


a 



€R" 
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Matrix T n is tridiagonal. It is also an example of a symmetric matrix that is 
Toeplitz (defined in a later problem). The characteristic polynomial of T n is 
p n (X) — det(XI n — T n ). Show that 



p n (X) = (k-a)p n -i{X) -b Pn-lCX) 



(4.P.5) 



for n = 3, 4, 5, Find p\(X), and /^M [initial conditions for the polyno- 
mial recursion in (4.P.5)]. 

4.20. Consider the following If: T e C nxn is Toeplitz if it has the form T — 

[ti-j]ij=o,i,..., n -i- Thus, for example, if n — 3 we have 



T = 



to t-l t-2 

h to f_i 
h h to 



Observe that in a Toeplitz matrix all of the elements on any given 
diagonal are equal to each other. A symmetric Toeplitz matrix has the 
form T — \t\i-j\\ since T — T T implies that t-i — ti for all i. Let x n — 
[x n ,o *n,\ • • • Xn,n-2 x n ,n-i] T € C . Let /„ € C" x " be the n x n exchange 
matrix (also called the contra- identity matrix) which is defined as the 
matrix yielding x n = J n x n = [*„,„- 1*„,„-2 • • ■ x n ,ix n fi] T , We see that /„ 
simply reverses the order of the elements of x n . An immediate con- 
sequence of this is that J^x n — x n (i.e., J% — /„). What is Jj? (Write 
this matrix out completely.) Suppose that T n is a symmetric Toeplitz 
matrix. 

(a) Show that (noting that i„ — J n r„) 



T n + 1 — 


In ^n 


= 


to 

. ?n 


r T 
T„ 


(nesting property). What is T„? 




(b) Show that 






^n^n^n : 


= T n 







(persymmetry property). 

(Comment: Toeplitz matrices have an important role to play in digital signal 
processing. For example, they appear in problems in spectral analysis, and 
in voice compression algorithms.) 

4.21. This problem is about a computationally efficient method to solve the linear 
system 



R n a n = o n eo, 



(4.P.6) 
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where R n e R" x " is symmetric and Toeplitz (recall Problem 4.20). All 
of the leading principle submatrices (recall the definition in Theo- 
rem 4.1) of R n are nonsingular. Also, eo = [1 • • • 0] T e R n , a n — 
[1 a n \ ■■■ a n , n -2 «n,n-i] r , and a% e R is unknown as well. Define 
e n — J n eo- Clearly, a n fi — 1 (all n). 



(a) Prove that 



R n a n = a^eo = o„e n , 



(4.P.7) 



where J„a„ = a„. 
(b) Observe that R n \_a n a n ] — cr^[eo e n ]. Augmented matrix [a n a n ] is 
n x 2, and so [eo e n ] is n x 2 as well. Prove that 



Rn 



a„ 
a„ 



In ole n 



(4.P.8) 



What is r\rP. (That is, find a simple expression for it.) 
(c) We wish to obtain 



R n +\[a n +\ a n+ i] — a n+l [eo e„+i] 



(4.P.9) 



from a manipulation of (4. P. 8). To this end, find a formula for parameter 
K„ € R in 



^H 



a n 
a„ 



1 K„ 

K n 1 



0„ eo f] n 
Vn ff„ 2 e„ 



1 tf» 

K n 1 



such that (4. P. 9) is obtained. This implies that we obtain the vector 
recursions 



a n +\ 







A"„ 







and 











Find the initial condition a\ e R lxl . What is er, 2 ? 



(d) Prove that 



ff„ +1 =ff„(l - /«:„). 



(e) Summarize the algorithm obtained in the previous steps in the form of 
pseudocode. The resulting algorithm is often called the Levinson-Durbin 
algorithm. 
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(f) Count the number of arithmetic operations needed to implement the 
Levinson-Durbin algorithm in (e). Compare this number to the num- 
ber of arithmetic operations needed by the general LU decomposition 
algorithm presented in Section 4.5. 

(g) Write a MATLAB function to implement the Levinson-Durbin algorithm. 
Test your algorithm out on the matrix 



2 1 
1 2 1 
1 2 



4.22. 



(Hint: You will need the properties in Problem 4.20. The parameters K n 
are called reflection coefficients, and are connected to certain problems in 
electrical transmission line theory.) 

This problem is about proving that solving (4. P. 6) yields the LDL T decom- 
position of R~ x . (Of course, R n is real-valued, symmetric, and Toeplitz.) 
Observe that (via the nesting property for Toeplitz matrices) 



Rn 



1 





■ 


0" 




[On 1 


X 


X 


X 


0/1,1 


1 


• 










<£l • 


X 


X 


0/1,2 


fl/z-1,1 


• 





= 








X 


X 


@n,n— 2 


a n -l,n-3 ■ 


1 













•• °l 


X 


Qn,n—1 


a n -l,n-2 ■ 


• 02,1 


1 _ 


, 


_ 





■■ 


°l\ 



= L„ 



= U„ 



(4.P.10) 



where x denotes "don't care" entries; thus, the particular value of such an 
entry is of no interest to us. Use (4.P.10) to prove that 



L n K n L n — L) n , 



(4.P.11) 



where D n — diag (cr„ , o^-v • • • ' a 2 > °t ) (diagonal matrix; see the comments 
following Theorem 4.4). Use (4. P. 11) to prove that 



r: 



l D J 



(4.P.12) 



What is D n l [Hint: Using (4.P.10), and R n — R^ we note that L T n R n L n can 
be expressed in two distinct but equivalent ways. This observation is used 
to establish (4.P.11).] 

4.23. A matrix T n is said to be strongly nonsingular (strongly regular) if all of its 
leading principle submatrices are nonsingular (recall the definition in Theo- 
rem 4.1). Suppose that x n — [x n q x n j • • • x„„_2 x nn _i\ T e C", and define 
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Z„ e C" x " according to 

Thus, Z„ shifts the elements of any column vector down by one position. 
The top position is filled in with a zero, while the last element x„„_i is lost. 
Assume that T n is real-valued, symmetric and Toeplitz. Consider 

t — 7 t y T — y 

(a) Find X n . 

(b) If T n is strongly nonsingular then what is rank (X n )l (Be careful. There 
may be separate cases to consider.) 

(c) Use &k (Kronecker delta) to specify Z„; that is, Z n — [zij], so what is 
Zij in terms of the Kronecker delta? (Hint: For example, the identity 
matrix can be described as / = [5,_ ; ].) 

4.24. Suppose that R e R NxN is strongly nonsingular (see Problem 4.23 for the 
definition of this term), and that R — R T . Thus, there exists the factorization 

R = L N D N L T N , (4.P.13) 

where Ln is a unit lower triangular matrix, and Dn is a diagonal matrix. 
Let 

L N = [/o h ■ ■ ■ In-2 In-i], 

so /,- is column i of L#. Let 

D N — diag(d , d\, . . . , d N -\)- 

Thus, via (4.P.13), we have 

N-\ 



R=Y J d Ml- (4.P-14) 



k=Q 

Consider the following algorithm: 

for n := to N - 1 do begin 
d n ■= eJ,R n e n \ 
In := d~ 1 R n e n ; 
ftn+1 := Rn — dnlnln ! 

end; 
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As usual, we have the unit vector 

e, = [0 • • • 1 



Of GR N . 



The algorithm above is the Jacobi procedure (algorithm) for computing the 
Chole sky factorization (recall Theorem 4.6) of R. 

Is R > a necessary condition for the algorithm to work? Explain. Test the 
Jacobi procedure out on 



R = 



2 1 
1 2 1 
1 2 



Is R > 0? Justify your answer. How many arithmetic operations are needed to 
implement the Jacobi procedure? How does this compare with the Gaussian 
elimination method for general LU factorization considered in Section 4.5? 



4.25. Suppose that A ' exists, and that / 
course, / is the identity matrix. 



V T A l U is also nonsingular. Of 
(a) Prove the Sherman-Morrison-Woodbury formula 



[A 



uv T r l 



A~ l U[I + v t a~ 1 u]~ 1 v t a- 



(b) Prove that if U = u e C" and V = v e C", then 

T , , A~ 1 uv T A~ 1 

(Comment: These identities can be used to develop adaptive filtering algo- 
rithms, for example.) 



4.26. Suppose that 



a b 
c 



Find conditions on a, b, and c to ensure that A > 0. 

4.27. Suppose that A € R" x ", and that A is not necessarily symmetric. We still 
say that A > iff x T Ax > for all x # (x e R"). Show that A > iff 
B — j(A + A T ) > 0. Matrix B is often called the symmetric part of A. (Note: 
In this book, unless stated to the contrary, a pd matrix is always assumed to 
be symmetric.) 
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4.28. Prove that for A e R" x " 



n-1 



|A||oo = max V|a/,/l 



0<i<»— 1 ' 



j=0 



4.29. Derive Eq. (4.128) (the law of cosines). 

4.30. Consider A„ e R" x " such that 



A n — 



A~ l = 



i 



— 1 

1 


-1 

-1 




— 1 — 1 

-1 -1 








1 




-1 -1 













1 -1 













1 


1 


1 2 




2"- 


-3 2 »-2 





1 1 




2"- 


-4 2 „-3 





1 




2"- 


-5 2 »-4 










1 


1 













1 



Show that Koo(A n ) — nl" l . Clearly, det(A„) = 1. Consider 

D„ = diag(10 _1 , 10" 1 , . . . , 10 _1 ) € R". 

What is K p (D n )7 Clearly, det(D„) = 10~". What do these two cases say 
about the relationship between det(A), and at (A) (A € R" x ", and is nonsin- 
gular) in general? 

4.31. Recall Problem 1.7 (in Chapter 1). Suppose a — [aoa\ ■ ■ -a n ] T , b — [bob\ 
■ ■ ■ b m ] T , and c — [c ci • • • c n+m ] T , where 



ci — ) y cikbi-k- 

k=Q 

Find matrix A such that c — Ab, and find matrix B such that c — Ba. What 
are the sizes of matrices A, and Bl {Hint: A and B will be rectangular 
Toeplitz matrices. This problem demonstrates the close association between 
Toeplitz matrices and the convolution operation, and so partially explains the 
central importance of Toeplitz matrices in digital signal processing.) 
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4.32. Matrix P e R" x " is a permutation matrix if it possesses exactly one one per 
row and column, and zeros everywhere else. Such a matrix simply reorders 
the elements in a vector. For example 



10 




xo 




x\ 


10 




XI 




X2 


10 




X2 




x 


1 




XT, 




XT, 



Show that P ' = P T (i.e., P is an orthogonal matrix). 
4.33. Find c and s in 



*0 
x\ 



"0 



l l 



where s 2 + c 2 — 1. 
4.34. Consider 



A = 



2 10 
12 10 
12 1 
12 



Use Householder transformation matrices to find the QR factorization of 
matrix A. 



4.35. Consider 



A = 



5 2 10 

2 5 2 1 
12 5 2 
12 5 



Using Householder transformation matrices, find orthogonal matrices Qq, 
and Q\ such that 



QiQoA = 



hoo hoi hoi hoi 

hio h\\ h l2 hi 3 

h 2 \ h 2 2 h 2 T, 

/?32 /J33 



= H. 



[Comment: Matrix H is upper Hessenberg. This problem is an illustration of 
a process that is important in finding matrix eigenvalues (Chapter 11).] 

4.36. Write a MATLAB function to implement the Householder QR factorization 
algorithm as given by the pseudocode in Section 4.6 [between Eqs. (4.146), 
and (4.147)]. The function must output the separate factors Hi that make up 
Q, in addition to the factors Q and R. Test your function out on the matrix 
A in Problem 4.34. 
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4.37. Prove Eq. (4.117), and establish that 

rank (A) = rank (A T ). 

4.38. If A € C" x ", then the trace of A is given by 

«-i 
Tr(A) = ^ a k,k 

(i.e., is the sum of the main diagonal elements of matrix A). Prove that 

||A|||=Tr(AA ff ) 

[recall (4.36a)]. 

4.39. Suppose that A, B e R" x ", and Q e R" x " is orthogonal; then, if A = 
QBQ T , prove that 

Tr(A) = Tr(fl). 

4.40. Recall Theorem 4.4. Prove that for A € R" x " 

\\m\ 2 f = y,°1 

k=0 

(Hint: Use the results from Problems 4.38 and 4.39.) 

4.41. Consider Theorem 4.9. Is a < 1 a necessary condition for convergence? 
Explain. 

4.42. Suppose that A € R" x " is pd and prove that the JOR method (Section 4.7) 
converges- if 



< &) < 



p{D- x A) 



4.43. Repeat the analysis made in Example 4.8, but instead use the matrix 



A = 



4 2 10 
14 10 
14 1 
2 4 



This will involve writing and running a suitable MATLAB function. Find the 
optimum choice for a> to an accuracy of ±0.02. 
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Orthogonal Polynomials 



5.1 INTRODUCTION 

Orthogonal polynomials arise in highly diverse settings. They can be solutions to 
special classes of differential equations that arise in mathematical physics problems. 
They are vital in the design of analog and digital filters. They arise in numerical 
integration methods, and they have a considerable role to play in solving least- 
squares and uniform approximation problems. 

Therefore, in this chapter we begin by considering some of the properties shared 
by all orthogonal polynomials. We then consider the special cases of Chebyshev, 
Hermite, and Legendre polynomials. Additionally, we consider the application 
of orthogonal polynomials to least-squares and uniform approximation problems. 
However, we emphasize the case of least-squares approximation, which was first 
considered in some depth in Chapter 4. The approach to least-squares problems 
taken here alleviates some of the concerns about ill-conditioning that were noted 
in Chapter 4. 



5.2 GENERAL PROPERTIES OF ORTHOGONAL POLYNOMIALS 

We are interested here in the inner product space L 2 (D), where D is the domain 
of definition of the functions in the space. Typically, D — [a, b], D — R, or D — 
[0, oo). We shall, as in Chapter 4, assume that all members of L 2 (D) are real- valued 
to simplify matters. So far, our inner product has been 

(f,g) = f f(x)g(x)dx, (5.1) 

J D 

but now we consider the weighted inner product 

(f,g)= f w(x)f(x)g(x)dx, (5.2) 

J D 

where w(x) > (x e D) is the weighting function. This includes (5.1) as a special 
case for which w(x) — 1 for all x e D. 



An Introduction to Numerical Analysis for Electrical and Computer Engineers, by C.J. Zarowski 
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Our goal is to consider polynomials 



<t>n(x) = Y^<Pn,kX k ,xe D (5.3) 

k=Q 

of degree n such that 

(<Pn,4>m) — K-m (5.4) 

for all n,m e Z + . The inner product is that of (5.2). Our polynomials {<p n (x)\n e 
Z + }. are orthogonal polynomials on D with respect to weighting function w(x). 
Changing D and/or w(x) will generate very different polynomials, and we will 
consider important special cases later. However, all orthogonal polynomials possess 
certain features in common with each other regardless of the choice of D or w(x). 
We shall consider a few of these in this section. 

If p(x) is a polynomial of degree n, then we may write dsg(p(x)) — n. 

Theorem 5.1: Any three consecutive orthogonal polynomials are related by the 
three-term recurrence formula {relation) 

<t>n+i(x) = (A n x + B n )(j) n (x) + C n (j)n-\{x), (5.5) 

where 

. _ 4>n+l,n+\ „ _ 4>n+l,n+\ ( <Pn+l,n <Pn,n-l 

A n — ; ^ "n — ; 



<Pn,n <Pn,n Wm + 1,m + 1 <Pn,n 

„ 0B+l,n+10«-l,n-l ,, „ 

C„ = - 2 . (5.6) 

Vn,n 

Proof Our proof is a somewhat expanded version of that in Isaacson and Keller 
[1, pp. 204-205]. 

Observe that for A n — (j> n +i,n+i/<Pn,n< we have 

h + 1 n 

q„(x) = <t> n +l(x) - A n X(j) n (x) = 'Y^(l>n+\,kX k ~ A„ ^ <t>n,kX k+l 

k=0 k=0 

n+\ n+\ 

= ^2<Pn+l,kX k ~ A n 'Y^ l 4>n,}-\X i 
fc=0 ;=1 

n n 

= ^24>n+l,kX k - A n 'Y^ l 4>n,j-\X i + [0 B+ i >B+ i - A n (/) ntn ]x n + l 
k=0 ;=1 

n n 

= ^2<Pn+l,kX k ~ A n 'Y^ l 4>n,}-\X } , 
k=0 j=\ 
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which is a polynomial of degree at most n, i.e., deg(q n (x)) < n. Thus, for suit- 
able Uk 



q n (x) = y^akfaW, 



k=0 

so because (fa, <j>j) — &k-j, we have a ; - = (q„, (j>j). In addition 

n-2 

0n+iW - A n xfa(x) = a„fa(x) + a n -\cj) n -i(x) + ^a k (j> k (x). (5.7) 

Now degC*0y(;c)) < « — 1 for j < n — 2. Thus, there are /8a such that 

n-i 
x0/(x) = ^Mk(x) 

k=0 

so (</>,-, x</) ; > = 0ifr>n — 1, or 

(fa,X<j>j} = (5.8) 

for j = 0, 1, ...,« — 2, From (5.7) via (5.8), we obtain 

{(/>„+! - A„X<p„,fa) — (<j>n+l,fa) - A n {(p n ,X<t>k) — 

for k — 0, 1, . . . , n — 2. This is the inner product of cj>k(x) with the left-hand side 
of (5.7). For the right-hand side of (5.7) 

/ "~ 2 \ 

= ( a„<p n + a n -\4>n-l + 2_, a j4>j' <Pk ) 

n-2 
= a n (<p n , fa) + a„-l {fa-l,fa) + /, &j (</>; , fa > 

7=0 

n-2 

= ^2 a .j(<i>j><i>k) 

again for £ = 0, . . . , n — 2. We can only have Yl'}Zo a j (4>j> <Pk) = if ay =0 for 
j = 0, 1, . . . , n — 2. Thus, (5.7) reduces to 

fa+i(x) - A n xfa{x) = a n fa(x) + a n -i<t>n-i(x) 

or 

<Pn+i(x) = (A„x + a„)0„(x)H-a„-i0„-i(x), (5.9) 
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which has the form of (5.5). We now need to verify that a n — B n , a n -\ — C„ as 
in (5.6). From (5.9) 

<p n (x) = A n -ix4> n -i(x) + a n -i4>„-i(x) + a n -2<t>n-2(x) 



or 



1 

x<f> n -i(x) = <p n (x) + 

A n -\ 



1 



-(a n -i(j>„-i(x) + a n - 2 (/>n-2(x)) 



(5.10) 



=P„-l(x) 

for which deg(/?„_i(x)) < n — 1. Thus, from (5.9) 

((pn+l, <l>n-\) — A n {x<j) n ,(t) n -i) +a n (</>„, (j)„-i) +a„-\ (</>„_i, </>„-i) 
so that 



and via (5.10) 



which becomes 



or 



a«-l = -A n (x<p n , 4>n-\) — -A n {(p n ,X<p n -\), 



a n -\ 



ln-1 



"0« + Pn-l ) , 



«»-l - ~ ; " (<t>n,<t>n) — 



Cm Of ,7 _ 1 



A n — \ A n — \ 

<l>n + l,n + l 4>n-l,n-l 



which is the expression for C n in (5.6). Expanding (5.5) 



«+i 



n+\ 



n-\ 



22&n+i,kx k — A n 22<p n , k -ix k + B n 22(j) n ^x k + C„ )J) n -\ :k x k , 

k=Q k=\ k=0 k=0 

where, on comparing the terms for k — n, we see that 

4 > n+l,n — A n (p n n -\ + D n <p nn 

(since 4> n -\,n =0). Therefore 



1 1 

B n — — [<Pn + l,n ~ A n <p nn -\] — — — 

tyn,n fin,) 



<Pn + l,n + l 
'«+!,« 7 <Pn,n-\ 



Pn+l,n+l 



"w + l,w H'n.n — 1 



,4>n+\,n+\ <Pn,n . 



which is the expression for B n in (5.6). 
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Theorem 5.2: Orthogonal polynomials satisfy the Christoffel-Darboux formula 

{relation) 

n , 

(x - y)Y](pj(x)<pj(y) = - — — — [(p n+1 (x)(/) n (y) - cj) n+l (y)(/) n (x)]. (5.11) 

~^ ' <Pn+\,n+\ 

Proof The proof is an expanded version of that from Isaacson and Keller [1, 
p. 205]. 

From (5.5), we obtain 

<M)O0m+iOO = (A n x + B n )(j) n (x)^ n (y) + C n cj) n -\{x)(i> n (y), (5.12) 

so reversing the roles of x and y gives 

4 l n{x)4> n+ i(y) = (A n y + B n )cp n {y)(j> n (x) + C„</>„_i(;y )</>„(*)■ (5.13) 

Subtracting (5.13) from (5.12) yields 

^n(y)4>n + l(x) - (/) n (x)^ n + i(y) = A n (x - y)(pn(x)<Pn(y) 

+ c n (<p n -i(x)(p„(y) - (Pn(x)<P n -i(y))- (5.14) 

We note that 



(5.15) 



C„ __ 0«-i,«-i __ 1 

A n <Pn,n A n —\ 

so (5.14) may be rewritten as 

(x - y)(/)n(x)(j)n(y) = A" 1 (</>„+] (x)(j>„(y) ~ <t>n+l{y)<t>n{x)) 

- A~\(<p n (x)<l> n -i(y) - (p n -i(x)<p n (y)). 
Now consider 

n n 

(x - y)J2 ( / ) j(x)cPj(y) = J2 {AfWj+iixWjW - <Pj+i(y)<t>j(x)] 

7=0 j=0 

- Ajl 1 [4>j(.x)4>j-i(y) - 4>j-i(x)<t>j(y)]] . (5.16) 

Since Aj — 0/+i j+i/<j>; j, we have AZi — <P-i,-i/(/>o,0 = because ^>-i,-i — 0. 
Taking this into account, the summation on the right-hand side of (5.16) reduces 
due to the cancellation of terms, 1 and so (5.16) finally becomes 



(X ~ y)^2(pj(x)(/>j(y) = - — - [<Pn+l(x)(t>n(y) ~ <t>n+l(y)4>n(x)], 

r- 
which is (5.11). 



,_ Pn + \,n + \ 



The cancellation process seen here is similar to the one that occurred in the derivation of the Dirichlet 
kernel identity (3.24). This mathematical technique seems to be a recurring theme in analysis. 
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The following corollary to Theorem 5.2 is from Hildebrand [2, p. 342]. However, 
there appears to be no proof in Ref. 2, so one is provided here. 

Corollary 5.1 With cp k (x) — dcf>k(x)/dx we have 

y>, 2 W = - <? "'" [0ffi (*)&,(*) - ti l) (x)<t> n+l (x)l (5.17) 

Proof Since 

<Pn+i(x)(j) n (y) - <p n+ i(y)(p n (x) = [0 n+ i(x) - 0„+i(y )]0„(y) 

- [0n(*) - 0n(>')]0n+l(}') 



via (5.11) 

n 

J^<l>j(x)<t>j(y) 



<Pn+l(x) ~4>n+l(y) , , , <Pn(x) - (j) n (y) 

0n(;y) 0«+i(y) 

'«+!,«+! L x - y x - y 



(5.18) 



Letting y — > x in (5.18) immediately yields (5.17). This is so from the definition of 
the derivative, and the fact that all polynomials may be differentiated any number 
of times. 

The three-term recurrence relation is a vital practical method for orthogonal 
polynomial generation. However, an alternative approach is the following. 

If deg(g r _i (x)) = r — 1 (q r -\ (x) is an arbitrary degree r — 1 polynomial), then 

(cj) r ,q r -l)— I w(x)(j) r (x)q r -i(x) dx — 0. (5.19) 

J D 

Define 

d r G r (x) /„■, 

w{x)<t> r {x)=—^- = G ( ;\x) (5.20) 

so that (5.19) becomes 

/ G { p(x)q r -i(x)dx = 0. (5.21) 

(V) 

If we repeatedly integrate by parts (using g^-iM = f° r a ^ x )' we obtain 

f Gj r) (x)«r-lWrf* = Gj r ~ 1) (x)?r-lW|D- f G^~ l) (x)q^ (x) dx 
J D J D 



TLFeBOOK 



GENERAL PROPERTIES OF ORTHOGONAL POLYNOMIALS 

= G< r - 1) G0«.-1(J0|D - G ( r 2 \x)q'; l \{x)\ D 
) (x)q r _ l (x)dx 

G < f- l \x)q r . x (x)\ D - G { ;- 2 \x)qf\{x)\ D 
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/. 



G (r-2)^_(2) 



G ( r 3) (x)qf-\{x)\ D + f G { r 3) (x)qf\{x)dx 
J D 



and so on. Finally 



<r-2)M) 



[G^ ^,-1 - g; '? r _j + g; ^ r _j 



,( r _3) (2) 



+ (-i) r - i G f .« r ( r 1 1) ]D = o, 



or alternatively 



Since from (5.20) 



Yy\f x G^\^ 



.k=\ 



= o. 



4>,-(x) = 



1 d r G r (;c) 

w(x) dx r 



(5.22a) 
(5.22b) 

(5.23) 



which is a polynomial of degree r, it must be the case that G,-{x) satisfies the 
differential equation 

d r+1 f 1 d r G r (x) 



dx r+1 



w{x) dx r 







(5.24) 



for x € D. Recalling D = [a, b] and allowing a — > — oo, & -> oo (so we may have 

(k) 

D = [a, oo), or D = R), Eq. (5.22a) must be satisfied for any values of q r _ x (a) and 



.(*) 



C ; i(*)(* = o,i, 



1). This implies that we have the boundary conditions 
Gf'(a) = 0,Gf ) (l)) = (5.25) 



for k = 0, 1, . . . , r — 1. These restrict the solution to (5.24). It can be shown [6] 
that (5.24) has a nontrivial 2 solution for all r e Z + assuming that w(x) > for 
all x e D, and that the moments f D x k w(x)dx exist for all k e Z + . Proving this 
is difficult, and so will not be considered. The expression in (5.23) may be called 
a Rodrigues formula for <p r (x) (although it should be said that this terminology 
usually arises only in the context of Legendre polynomials). 

The Christoffel-Darboux formulas arise in various settings. For example, they 
are relevant to the problem of designing Savitzky-Golay smoothing digital filters 
[3], and they arise in numerical integration methods [2, 4]. Another way in which 
the Christoffel-Darboux formula of (5.11) can make an appearance is as follows. 

The trivial solution for a differential equation is the identically zero function. 
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We may wish to approximate f(x) e L 2 (D) as 

n 

fix) « ^djcpjix), 



so the residual is 



r„O0 = /(*) - ^aj<t>j(x). 



(5.26) 



(5.27) 



Adopting the now familiar least-squares approach, we select a; to minimize func- 
tional V(a) = \\r n \\ 2 = (r n ,r n ) 



fix) - ^2,a.j(j>jix) 



Via) = / w(x)r n (x)dx — J w(x) 
J d J D 

where a = [ao a\ ■ ■ ■ a n ] T € R" +1 . Of course, this expands to become 

V(, 



dx, (5.28) 



(a) — J w(x)f (x)dx — 2 2_, a ] I w(x)f(x)(j)j(x)dx 

JD ;_ n J D 

n n . 

+ J2J2 a J ak / 



;'=o 

w(x)4>j(x)<pk(x) dx, 



;=0 k=Q 



so if g - [go gi ■■■ g n ] T with gj = | D w(x)/(x)0 7 (x) rfx, and /? = [r,j] € 
R («+i)x(«+i) wim r> _ = j^ w ( x )<t>i(x)<t>jix)dx, and p = f D w(x)f 2 (x)dx, then 



y(a) = p — 2a g + a Ra. 



(5.29) 



But r,- ; = (<pi, <pj) — Si-j [via (5.4)], so we have R — I (identity matrix). Imme- 
diately, one of the advantages of working with orthogonal polynomials is that R 
is perfectly conditioned (contrast this with the Hilbert matrix of Chapter 4). This 
in itself is a powerful incentive to consider working with orthogonal polynomial 
expansions. Another nice consequence is that the optimal solution a satisfies 



a = g, 



that is 



aj = / w{x)f{x)(j)j(x)dx = (/, <j)j), 

JD 

where j e Z„+i. If a — a in (5.27) then we have the optimal residual 



f n ix) = fix) - ^2aj(j)j(x). 
;=0 



(5.30) 
(5.31) 

(5.32) 
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We may substitute (5.31) into (5.32) to obtain 



n r- 



r n (x) = f(x)-l^ J w(y)f{y)<t>j{y)dy 



<Pj(x) 



L 



fix)- / f(y)w(y) 



J2<Pj(x)<Pj(y) 



dy. 



(5.33) 



For convenience we define the kernel function 

n 

K n (x,y) = ^2^)j{x)(pj{y), 

so that (5.33) becomes 

rn(x) = f(x)- w(y)f(y)K n (x,y)dy. 



(5.34) 



(5.35) 



Clearly, K n (x, y) in (5.34) has the alternative formula given by (5.11). Now con- 
sider 



f f(x)w(y)K n (x,y)dy = f(x) f W^Tcpj 
Jd Jd j=0 ' 



(x)(pj(y)dy 



n 

= f(x)Y](pj(x) w(y)cpj(y)dy 
.7=0 ' Jd 

n 

= f(x)Y i <t>j(x){l,4>j) 

.7=0 

El d>o(x) 



.7=0 



</>0,0 



d>o(x) 
= fix)^ 1 = f(x) 

00,0 

because 4>q(x) — (/>o,o for x e D, and (0o, 4>j) — $j- Thus, (5.35) becomes 
f n ix) = / /(x)wO0^h(*,)0 dy - I f(y)w(y)K n (x,y)dy 

J D J D 



L 



wiy)K n (x,y)[fix)-fiy)]dy, 



(5.36) 



Recall the Dirichlet kernel of Chapter 3, which we now see is really just a special instance of the 
general idea considered here. 
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The optimal residual (optimal error) f n (x) presumably gets smaller in some sense 
as n — > oo. Clearly, insight into how it behaves in the limit as n -> oo can be 
provided by (5.36). The behavior certainly depends on f(x), w(x), and the kernel 
K n (x,y). Intuitively, the summation expression for K n (x,y) in (5.34) is likely 
to be less convenient to work with than the alternative expression we obtain 
from (5.11). 

Some basic results on polynomial approximations are as follows. 

Theorem 5.3: Weierstrass' Approximation Theorem Let f(x) be continu- 
ous on the closed interval [a, b]. For any e > there exists an integer N — N(e), 
and a polynomial pn(x) € P [a, b] (deg(/?w(x)) < N) such that 

\f(x)-p N (x)\ < e 

for all x e [a, b]. 

Various proofs of this result exist in the literature (e.g., see Rice [5, p. 121] and 
Isaacson and Keller [1, pp. 183-186]), but we omit them here. We see that the 
convergence of pn(x) to f(x) is uniform as N — > oo (recall Definition 3.4 in 
Chapter 3). Theorem 5.3 states that any function continuous on an interval may be 
approximated with a polynomial to arbitrary accuracy. Of course, a large degree 
polynomial may be needed to achieve a particular level of accuracy depending upon 
f(x). We also remark that Weierstrass' theorem is an existence theorem. It claims 
the existence of a polynomial that uniformly approximates a continuous function 
on [a, b], but it does not tell us how to find the polynomial. Some information 
about the convergence behavior of a least-squares approximation is provided by 
the following theorem. 

Theorem 5.4: Let D — [a, b], w(x) — 1 for all x e D. Let f(x) be continuous 
on D, and let 

n 

qn(x) = ^fl,(/);(x) 

(least-squares polynomial approximation to f(x) on D), so cij — (f, <pj) = 
f a f(x)<t>j(x)dx. Then 

lim V(a)= lim / [f(x) — q n (x)] 2 dx — 0, 

and we have Parseval's equality 

rb °° 

/ f 2 (x)dx^J2 a r < 5 - 37 ) 



J 



=0 
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Proof We use proof by contradiction. Assume that linin^oo V(a) — S > 0. 
Pick any e > such that e 2 — 2 (b-a) ^- ^ Theorem 5.3 (Weierstrass' theorem) 
there is a polynomial p m (x) such that \f(x) — p m (x)\ < € for x e D. Thus 



/' 

J a 



[f (x) - p m {x)f dx < e 2 [b-a]= -8. 



Now via (5.29) and (5.30) 
that is 



V(a) = p — a a, 



/ " 

f 2 (x)dx -^2,a 2 , 

;'=o 
and V(a) > for all a e R" +1 . So we have Bessel's inequality 

rb " 

a) = / f 2 (x)dx-J2tf ^ °- 



6 
V(a) = / / z (x)Jx-^«t ^ °. ( 5 - 38 ) 



and V(a) must be a nonincreasing function of n. Hence the least- squares approxi- 
mation of degree m, say, q m (x), satisfies 



-S> f [f(x)-q m (x)?dx>8 

*■ J a 



which is a contradiction unless 5 = 0. Since 5 = 0, we must have (5.37). 

We observe that Parseval's equality of (5.37) relates the energy of f(x) to 
the energy of the coefficients a,-. The convergence behavior described by Theo- 
rem 5.4 is sometimes referred to as convergence in the mean [1, pp. 197-198]. 
This theorem does not say what happens if f(x) is not continuous on [a, b]. The 
result (5.36) is independent regardless of whether f(x) is continuous. It is a poten- 
tially more powerful result for this reason. It turns out that the convergence of 
the least-squares polynomial sequence (q n (x)) to f(x) is pointwise in general but, 
depending on </>/(x) and f(x), the convergence can sometimes be uniform. For uni- 
form convergence, f(x) must be sufficiently smooth. The pointwise convergence 
of the orthogonal polynomial series when f(x) has a discontinuity implies that the 
Gibbs phenomenon can be expected. (Recall this phenomenon in the context of the 
Fourier series expansion as seen in Section 3.4.) 

We have seen polynomial approximation to functions in Chapter 3. There we saw 
that the Taylor formula is a polynomial approximation to a function with a number 
of derivatives equal to the degree of the Taylor polynomial. This approximation 
technique is obviously limited to functions that are sufficiently differentiable. But 
our present polynomial approximation methodology has no such limitation. 
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We remark that Theorem 5.4 suggests (but does not prove) that if f(x) € 

L 2 [a, b], then 

oo 

fM = J2{f,<Pj)<t>j(x), (5.39) 

.7=0 

where 

f2> 



(f,4> 



j)= f f(x)(Pj(x)dx. (5.40) 

J a 

This has the basic form of the Fourier series expansion that was first seen in 
Chapter 1. For this reason (5.39) is sometimes called a generalized Fourier series 
expansion, although in the most general form of this idea the orthogonal functions 
are not necessarily always polynomials. The idea of a generalized Fourier series 
expansion can be extended to domains such as D — [0, oo), or D — R, and to any 
weighting functions that lead to solutions to (5.24). 

The next three sections consider the Chebyshev, Hermite, and Legendre poly- 
nomials as examples of how to apply the core theory of this section. 



5.3 CHEBYSHEV POLYNOMIALS 

Suppose that D = [a, b] and recall (5.28), 

rb 

V{a)= / w{x)rl{x)dx, (5.41) 

J a 

which is the (weighted) energy of the approximation error (residual) in (5.27): 

n 

r n {x) = f{x)-Y j a j <t> j (x). (5.42) 

7=0 

The weighting function w(x) is often selected to give more or less weight to errors 
in different places on the interval [a,b]. This is intended to achieve a degree of 
control over error behavior. If w(x) — c > for x e [a, b], then equal importance 
is given to errors across the interval. This choice (with c — 1) gives rise to the 
Legendre polynomials, and will be considered later. If we wish to give more weight 
to errors at the ends of the interval, then a popular instance of this is for D = [— 1, 1] 
with weighting function 

w(x) = 1 . (5.43) 

VI — x l 

This choice leads to the famous Chebyshev polynomials of the first kind. The reader 
will most likely see these applied to problems in analog and digital filter design in 
subsequent courses. For now, we concentrate on their basic theory. 
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The following lemma and ideas expressed in it are pivotal in understanding the 
Chebyshev polynomials and how they are constructed. 

Lemma 5.1: If n e Z + , then 

n 
cos nO = Y^ Pn,k cos* 9 (5.44) 

k=0 

for suitable ft n ^ g R. 

Proof First of all recall the trigonometric identities 

cos(m + 1)9 = cos m.9 cos 9 — sin m.9 sin 9 
cos(m — 1)9 = cos m0 cos + sin m0 sin 

(which follow from the more basic identity cos(a + b) = cos a cos£> — sin a sinfc). 
From the sum of these identities 

cos(m + 1)9 = 2 cos m9 cos — cos(m — 1)9. (5.45) 

For n = 1 in (5.44), we have /Jio = 0, /3i,i = 1, and for n = 0, we have /3o,o = 
1. These will form initial conditions in a recursion that we now derive using 
mathematical induction. 

Assume that (5.44) is valid both for n = m and for n = m — 1, and so 

m 

cos m9 = 2_,Pm,k cos 

k=0 
m—\ 



cos(m — \)9 = yj Pm-\,k cos 



jfc=0 

and so via (5.45) 

m m — 1 



COS(m + 1)0 = 2C0S# \]Pm,k COS — >_^ Pm-l,k COS 
fc=0 &=0 

m m— 1 

= 2 ^ £„,,* COS* +1 - J2 Pm-l,k COS* 
k=0 k=0 

m + \ m—\ 

— 2 2_j Pm,r-\ COS r — 2_^ fi m -\ : r COS'" 



r=l r=0 
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m+1 



= y^[2j6 mife _i - m -i,fc]cos fe 0(0 m ,* — for k < 0, and k > m) 

k=Q 
m+1 



jt=0 

which is to say that 



-1,* = 20 m ,fc-l — 0m-l,*> (5.46) 



This is the desired three-term recursion for the coefficients in (5.44). As a conse- 
quence of this result, Eq. (5.44) is valid for n — m + 1 and thus is valid for all 
n > by mathematical induction. 

This lemma states that cos n0 may be expressed as a polynomial of degree n in 
cos0. Equation (5.46) along with the initial conditions 

A>.0 = 1. 01,0 = 0, U = 1 (5.47) 

tells us how to find the polynomial coefficients. For example, from (5.46) 

02,* = 201,fc_l - /Jo,*: 

for £ = 0, 1, 2. [In general, we evaluate (5.46) for k = 0, 1, . . . , m + 1.] Therefore 

02.0 = 20i,-i — 00,0 = — 1, 

02.1 = 20i,o — 00,1 = 0, 

02.2 = 20i,i — 00,2 = 2 

which implies that cos 26 — — 1 + 2 cos 2 0. This is certainly true as it is a well- 
known trigonometric identity. 

Lemma 5.1 possesses a converse. Any polynomial in cos0 of degree n can be 
expressed as a linear combination of members from the set {cos k0 \k = 0, 1, . . . , n). 

We now have enough information to derive the Chebyshev polynomials. Recall- 
ing (5.19) we need a polynomial (p r (x) of degree r such that 



/ 



1 1 

c/) r (x)q r -i(x)dx = 0, (5.48) 



vT^? 



where q,--\{x) is an arbitrary polynomial such that deg(g r -i(x)) < r — 1. Let us 
change variables according to 

jt = cos(9. (5.49) 
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Then dx — — smO dO, and x e [— 1, 1] maps to 6 e [0, it]. Thus 



f l 1 [° I 

/ d> r (x)q r -i(x) dx = - I - 

i-i VI- x l Jjt VI— cos 2 

= / (J) r (cos0)q r - l (cos6)d6 = 
Jo 



(cos#)gy_i(cos#) sin0d0 



Jo 



Because of Lemma 5.1 (and the above mentioned converse to it) 

, (cosd)coskOdO = 
for k = 0, 1, . . . , r — 1. Consider </v(cos#) = C,- cosr0, then 

r 

[cos(r + &)6> + cos(r - fc)0] J0 



(5.50) 



(5.51) 



Jo 



C r I cos r6 cos k6 d6 

/o 2 



1 f* 



1 



1 1 

sin(r + k)9 H sin(r - k)B 

r + k r — k 



= 



Jo 



for k = 0, I, . . . , r — 1. Thus, we may indeed choose 



t> r (x) — C r cos[r cos x]. 



(5.52) 



Constant C r is selected to normalize the polynomial according to user requirements. 
Perhaps the most common choice is simply to set C r — 1 for all r e Z + . In this 
case we set T r (x) — 4> r (x): 



T r {x) — cos[r cos x]. 

These are the Chebyshev polynomials of the first kind. 
By construction, if r ^ k, then 



(5.53) 



(T r , T k ) 



L VT=x* Tr 



(x)Tk(x) dx — 0. 



Consider r — k 



= f — 



-T (x) dx 



cos [r cos x]dx, 



and apply (5.49). Thus 



WTrW 



/ cos 2 rd d$ = I 



n, r = 
j7T, r > 



(5.54) 



(5.55) 
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We have claimed <j> r (x) in (5.52), and hence T r (x) in (5.53), are polynomials. This is 
not immediately obvious, given that the expressions are in terms of trigonometric 
functions. We will now confirm that T r (x) is indeed a polynomial in x for all 
r e Z + . Our approach follows the proof of Lemma 5.1. 
First, we observe that 

Tq(x) — cos(0 • cos - x) — 1, T\(x) — cos(l • cos - x) — x. (5.56) 

Clearly, these are polynomials in x. Once again (see the proof of Lemma 5.1) 

T n+ \{x) = cos(n + 1)9(9 — cos - x) 

— cosn9 cos0 — (cos(« — 1)9 — cosn9 cos9) 

— 2 cos n9 cos 9 — cos(« — 1)9 
= 2xT n (x) - T n -i(x), 

that is 

T„+i(x) — 2xT n (x) - T n -i(x) (5.57) 

for n e N. The initial conditions are expressed in (5.56). We see that this is a 
special case of (5.5) (Theorem 5.1). It is immediately clear from (5.57) and (5.56) 
that T n (x) is a polynomial in x for all n e Z + . 

Some plots of Chebyshev polynomials of the first kind appear in Fig. 5.1. We 
remark that on the interval x € [— 1, 1] the "ripples" of the polynomials are of 
the same height. This fact makes this class of polynomials quite useful in certain 
uniform approximation problems. It is one of the main properties that makes this 
class of polynomials useful in filter design. Some indication of how the Chebyshev 
polynomials relate to uniform approximation problems appears in Section 5.7. 

Example 5.1 This example is about how to program the recursion specified 
in (5.57). Define 

n 
.7=0 

so that T„j is the coefficient of x J . This is the same kind of notation as in (5.3) 
for 4> n (x). In this setting it is quite important to note that T n j — for j < 0, and 
for j > n. From (5.57), we obtain 

n+l n n—\ 

/ , ln+\,jX = t~X > ^ 'n,jX / j ln — \jX' , 

j=0 .7=0 .7=0 



and so 



n+l n n—\ 

7=0 7=0 .7=0 
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Figure 5.1 Chebyshev polynomials of the first kind of degrees 2, 3, 4, 5. 

Changing the variable in the second summation according to k — j + 1 (so j 
k — 1) yields 

n + \ n+\ n — \ 

k=0 k=\ k=0 

and modifying the limits on the summations on the right-hand side yields 



n + 1 



n + 1 



n + 1 



7 , T n +i,kX =2/, T n: k-ix — 2_^ T n -\,kX ■ 



fc=0 



jt=0 



)t=0 



We emphasize that this is permitted because we recall that T n -\ — T n -\,n — 
T n -\^ n+ \ — for all n > 1. Comparing like powers of x gives us the recursion 

T n +\,k — 1Tn,k-\ — T n -\,k 

for k = 0, 1, . . . , n + 1 with n — 1, 2, 3, ... . Since Tq(x) — 1, we have Too — 1, 
and since 7\(x) = *, we have 7^0 = 0, 7ij = 1, which are the initial conditions 
for the recursion. Therefore, a pseudocode program to compute T n (x) is 

7b,o:=1; 

7"1,0:=0;7"1,1 :=1; 
for n := 1 to A/ - 1 do begin 
for /f := to n + 1 do begin 
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T n+1,k '■= 2 T n,k-1 ~ T n--\,k'< 
end; 
end; 



This computes T 2 (x), 



Tff(x) (so we need N > 2). 



We remark that the recursion in Example 5.1 may be implemented using integer 
arithmetic, and so there will be no rounding or quantization errors involved in 
computing T n (x). However, there is the risk of computing machine overflow. 

Example 5.2 This example is about changing representations. Specifically, 
how might we express 

n 
Pn(x)=J^Pn,jX j eP"[-l,l] 

.7=0 

in terms of the Chebyshev polynomials of the first kind? In other words, we wish 
to determine the series coefficients in 

n 

Pn(x) = ^ajTjix). 

j=o 

We will consider this problem only for n — 4. 

Therefore, we begin by noting that (via (5.56) and (5.57)) Tq(x) — 1, T\(x) — x, 
T 2 (x) — 2x 2 — 1, Tj,{x) — 4r 3 — 3x, and T^(x) — 8x 4 — 8x 2 + 1. Consequently 



/ ctjTjix) — ao + u\x + 2«2X 

7=0 



a 2 



+ 4a3JC — 3a^x + 8«4X — 8014X + oa\ 
— (ao — <*2 + u/\) + (<X\ — 3a^,)x + (2a 2 — 8a4)x 
implying that (on comparing like powers of x) 

P4,o — a -a 2 +a 4 , p^\ = a\ - 3«3, 
P4,2 = 2a2 — 8a4, 7743 = 4a3, 7744 = 8a4. 

This may be more conveniently expressed in matrix form: 



- 4a 3 x 3 + 8a 4 x 4 



1 





-1 





1 






1 






2 


-3 




-8 











4 

















8 



a 




/>4,0 


a\ 




P4,l 


a 2 


= 


P4,2 


«3 




P4,3 


Q!4 




PAA 
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The upper triangular system Ua — p is certainly easy to solve using the backward 
substitution algorithm presented in Chapter 4 (see Section 4.5). We note that the 
elements of matrix U are the coefficients [T n j] as might be obtained from the 
algorithm in Example 5.1. 

Can you guess, on the basis of Example 5.2, what matrix U will be for any n ? 



5.4 HERMITE POLYNOMIALS 

Now let us consider D — R with weighting function 

w{x) = e- alx \ (5.58) 

This is essentially the Gaussian pulse from Chapter 3. Recalling (5.23), we have 



„2 V 2 d r G r (x) 
t>r(x) = e a x —pf 1 , (5.59) 

dx r 



where G r (x) satisfies the differential equation 

d r+: 



dx r+l 



a 2 x 2 d r G r (x) 
dx r 



(5.60) 



[recall (5.24)]. From (5.25) G,-(x) and Gf\x) (for k = 1, 2, . . . , r - 1) must tend 
to zero as x -> ±oo. We may consider the trial solution 

G r (x) = Cre'" 2 * 2 . (5.61) 

The kth derivative of this is of the form 

_ 2 2 

C r e a x x (polynomial of degree k), 
so (5.61) satisfies both (5.60) and the required boundary conditions. Therefore 

rW = C r e« V / 7 ( e - aV ). (5.62) 

dx r 

It is common practice to define the Hermite polynomials to be 4> r (x) for C r — 
(—iy with either a 2 — 1 or a 2 — j. We shall select a 2 — 1, and so our Hermite 
polynomials are 

H r {x) = {-\) r e x2 ^- r {e-* 2 ). (5.63) 

By construction, for k ^ r 



f 



e~ a x (P r (x)<t> k (x) dx = 0. (5.64) 

For the case where k — r, the following result is helpful. 



TLFeBOOK 



226 ORTHOGONAL POLYNOMIALS 

It must be the case that 



-(*) = X!^ r 



k x 



k=0 



so that 



H0 r || 2 = / w(x)<fi(x)dx = / W(x)(j) r 

J D J D 



(x) 



Y, &.*** 



.k=Q 



dx, 



but f D w(x)<p r (x)x l dx = for i = 0, 1, . . . , r — 1 [special case of (5.19)], and so 
||0 r || 2 = 0,. r / w(x)4> r (x)x r dx = </>,-, r \ x r G^ (x) dx (5.65) 

J D J D 

[via (5.20)]. We may integrate (5.65) by parts r times, and apply (5.25) to obtain 

(5.66) 



\\<t>r\\ 2 = (-V r r\<t>r,r I G,-(x)dx. 
J D 

So, for our present problem with k — r, we obtain 



/oo 
e- av tf(x)dx = (-l) r 
-oo 



/oo 
C r e _a * dx. (5.67) 

-OG 



Now (if y — ax with a > 0) 

1"°° 2 2 f 00 2 2 If 00 ! 1 

/ e~ a x dx = 2 e~ a x dx = - / e"- v <fy = -y^F 

./-OO i0 « ^0 a 

via (3.103) in Chapter 3. Consequently, (5.67) becomes 



1101-11 2 = (-iyr\4>r,rC r 



(5.68) 



(5.69) 



With C r — (— l) r and a = 1, we recall H r (x) — <j> r {x), so (5.69) becomes 



\\H r \\ 2 = r\H r , r ^. 



(5.70) 



We need an expression for H rr . 
We know that for suitable p n ^ 



d" _ x i _ 
dx"* 



J^P" 



kx 



Lk=0 



(5.71) 



-p„(x) 
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Now consider 



d n+l 



dx n+l 



dx 



-2x 



-k=Q 
n 

.k=0 
n+\ 



k X 



kX 



^2, kPn ' kX 



k-\ 



\-k=\ 



"2 ^2 Pn,k-\ + ^(k + l)Pn,k+l 

k=0 k=0 

1 
= ^2 V-?-Pn,k-\ + (k+ l)Pn,k+l] X k e 



x k e-* 2 



-Pn+l,k 



k=Q 

so we have the recurrence relation 

Pn+l,k — -2Pn,k-l + (k + l)p n ,k+l- 

From (5.71) and (5.63) H n (x) = (-1)" p n (x). From (5.72) 

Pn + l,n + l = -2pn,n + (« + 2)/?„, n+ 2 = -2p n ,n, 

and so 

fln+l.n+l — ( — 1) Pn+l,n+\ — — (— 1) ( — 2Pn,n) — 2//,,,„. 



(5.72) 



(5.73) 



Because Hq(x) — 1 [via (5.63)], //o,o = 1, and immediately we have //„„ = 2" 
[solution to the difference equation (5.73)]. Therefore 



(5.74) 



\\H,Y = 2 r rW7T 



thus, f^ e- x2 H?(x) dx = 2 r r ly^T. 

A three-term recurrence relation for H n (x) is needed. Define the generating 
function 

(5.75) 



S(x, t) — exp[-f 2 + 2xt] — exp[x 2 - (t - x) 2 ]. 



Observe that 

3" 3" 3" 

— S(x, t) = exp[x 2 ]— exp[-(r - x) 2 ] = (-l)"exp[x 2 ] — exp[-(f - x) 2 ]. 

(5.76) 
Because of the second equality in (5.76), we have 

S (n \x, 0) = —-S(x, 0) = (-l)"exp[x 2 ] — exp[-x 2 ] = H n (x). (5.77) 

dt" dx" 
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The Maclaurin expansion of S(x, t) about t — is [recall (3.75)] 



n=0 



so via (5.77), this becomes 

* — ' w! 

«=0 

Now, from (5.75) and (5.78), we have 

M = 2fe -' 2 +^ = f; ^ff„ W , (5.79a) 



Unix), 



dx '—' n\ 

n=0 



and also 



dS ^t"dH n (x) ^ t n+1 dH n+l (x) /c ^ nu ^ 

— = > = > . (5.79b) 

dx ^ n\ dx ^ (n+ 1)! dx 

n=0 n=—\ 

Comparing the like powers of t in (5.79) yields 

1 dH n+l (x) 2 

= — H n (x), 



(n+1)! dx n\ 

which implies that 



dx 

for n e Z + . We may also consider 



dH n+ \ (x) 

= 2(n + l)H„(x) (5.80) 



35 »2_i_t,» x— * (—2t + 2x) „ 

— = {-It + 2x)e- t +2,x = V ■ -t n H n (x) (5.81a) 

rit ' * n 



and 



dt 

n=0 



r) S °° *^ — 1 °° *rt 

^7 = E? TTT flr » (x) = E-T ff »+ 1 ^ ) - (5 - 81b) 

at *-^ (n — 1)! *-^ /i! 

From (5.81a) 

V i ■ -t n H n {x) = -2 V — H n -i(x) + 2x V -#„(*). (5.82) 

■^^ «! *— ' (ra — 1)! ^^ n\ 

n=0 n=0 n=0 
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Figure 5.2 Hermite polynomials of degrees 2, 3, 4, 5. 



2.5 



Comparing like powers of t in (5.81b) and (5.82) yields 

1 2 1 

— H n+ i(x) = — -H n -i(x) + 2x — H„(x) 

n\ (n — 1)! n\ 



or 



H n +i(x) — 2xH n (x) - 2nH n -i(x). 



(5.83) 



This holds for all n € N. Equation (5.83) is another special case of (5.5). 

The Hermite polynomials are relevant to quantum mechanics (quantum harmonic 
oscillator problem), and they arise in signal processing as well (often in connection 
with the uncertainty principle for signals). A few Hermite polynomials are plotted 
in Fig. 5.2. 



5.5 LEGENDRE POLYNOMIALS 

We consider D = [— 1, 1] with uniform weighting function 

w(x) = 1 for all x e D. 



(5.84) 
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In this case (5.24) becomes 

,/H-i r 



dx r+l 



d r G r (x) 
~dx r 



= 



d 2r+1 



-G r (x) = 0. 



dx 2r + l 
The boundary conditions of (5.25) become 

G®(±1) = 

for k e Z r . Consequently, the solution to (5.85) is given by 



Thus, from (5.23) 



G r (x) = C,-(x 2 - l) r . 



t> r {x) = c r — (x 2 -iy. 



The Legendre polynomials use C r = jf^t, ar, d are denoted by 

1 d r 



Prix) = 



2 r r\dx' 



-(x'-iy, 



(5.85) 



(5.86) 



(5.87) 



(5.88) 



(5.89) 



which is the Rodrigues formula for P r (x). By construction, for k ^ r, we must 
have 



/> 



(x)P k (x)dx = 0. 



(5.90) 



From (5.66) and (5.87), we obtain 

,2 (-I)'' 



\\p r \r 



Recalling the binomial theorem 



IY /■! 

— P rr / (x 2 - l) r dx. 
r ' J-i 



(5.91) 



(a + fc) r = J] ( I ) «^ r " 



(5.92) 
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we see that 



<1r r dx T L ~ l \ k 

k=Q 



dx r 



1 \-\y- k x 2k 



d r 
dx r 



,2r 



r-\ 

E 

k=0 



I )(-iy- k x 2k 



= 2r(2r-l)---(r + l)x r + — 

dx r 



>-l 

£ 

.k=0 



I )(-iy- k x 2k 



implying that (recall (5.89)) 



Pr.r 



2r(2r - 1) • • • (r + 1) (2r)\ 



2 r r\ 



2 r [r\] 2 ' 



(5.93) 



But we need to evaluate the integral in (5.91) as well. With the change of variable 
x — sin 9, we have 



/l rrc/2 

(x 2 - \y dx = {-xy \ cos 2r+1 0do. 
-1 J-TZl 



(5.94) 



T/2 



We may integrate by parts 



/Tt/A fit, 

cos 2 ^ 1 9 d0 = / 
-7T/2 J-jr 



7T/2 



cos 6» cos 2r 6> J6» 



/2 



- cos 2r sin 6\ n/2 n + 2r I cos 2r_1 9 sin 2 (9 </6> 

' T/2 



/■7T/2 

J -it II 



(in f u dv — uv — f v du, we let w = cos 2r 9, and <iu = cos 9 dO), which becomes 
(on using the identity sin 2 6 = 1— cos 2 9) 



/7t/2 £71(2 fit/2 

cos 2 ^ 1 9d9 =2r / cos 2r - 1 9 d9 - 2r \ cos 2r+1 9 d6 
-7T/2 J-lc/2 J-7ll 



T/2 



for r > 1. Therefore 



/Itjl <-JT ; 

cos 2r+1 9d9 =2r 
-71/2 J-TZ 



(2r + 1) / cos 2r+1 rf(9 



7T/2 

/2 



cos 2r " 1 0c/0. 



(5.95) 



Now, since [via (5.94)] 



/ r _i=-(-l) 



/■T/2 

J-tt/2 



cos 2r - 1 9 d9, 
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Eq. (5.95) becomes 

(2r + l)(-l) r I r = -2r(-iy7 r _i, 
or more simply 



2r 



2r + 1 



Ir-l- 



(5.96) 



This holds for r > 1 with initial condition Iq — 2. The solution to the difference 

equation is 

2 2r+1 [r!] 2 
/,. = (-l) r — !_, (5.97) 

^ ' (2r + l)! 

This can be confirmed by direct substitution of (5.97) into (5.96). Consequently, if 
we combine (5.91), (5.93), and (5.97), then 

, (-1) 2 (2r)! (-l) r 2 2r+1 [r!] 2 
Il^|| 2 = 



which simplifies to 



2 r 2 r [r!] 2 (2r + l)! 

? 2 

ll^ll 2 



2r + 1 



(5.98) 



A closed-form expression for P n (x) is possible using (5.92) in (5.89). Specifi- 
cally, consider 



P„(x) 



i in i jb 

1 -(x 2 -ir 



2»M!fif:c" 



2"re! djc" 



E( 

.k=0 v 7 



(_l)* x 2»-2* 



(5.99) 



where M — n/2 (re even), or M = (n — l)/2 (re odd). We observe that 

d" 



-2k 



__ x m-zk = (2n - 2k)(2n - 2k - 1) • • • (re - 2k + l)x n ~ ZK . (5.100) 



Now 



(2n - 2fc)! = (2re - 2fc)(2re - 2fc - 1) • • • (re - 2k + 1) (re - 2fc) • • • 2 • 1, 

so (5.100) becomes 



=(n-2Jt)! 



il JC 2„-2* = (2n-2fc)! ^,_ 2 , 
Jjc" " (n-2k)\ 



(5.101) 



Thus, (5.99) reduces to 



,.«=^i;(;W^m- 



2"re! ^\ i 
fc=0 v 



(n-2k)\ 
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or alternatively 

/>„(*) = yVD* (2 "~ 2fe)! x"~ 2k . (5.102) 

Z ^ 2 n k\(n-k)\(n-2k)\ 

Consider /(*) = -±= so that /(*>(*) = (2*-D(2*-3)-3-i (1 _ x) -(2fc+i)/2 (for 
fc> 1). Define (2fc - 1)H = (2* - 1)(2£ - 3) • • • 3 • 1, and define (-1)!! = 1. As 
usual, define 0! = 1. We note that (2n)\ = 2"n\(2n — 1)!! which may be seen by 
considering 

(2«)! = (2n)(2n - l)(2n - 2)(2n - 3) • • • 3 • 2 • 1 

= [(2n)(2n - 2) ■ ■ ■ 4 ■ 2][(2n - 1)(2« - 3) • • • 3 • 1] = 2"«!(2« - 1)!!. 

(5.103) 
Consequently, the Maclaurin expansion for f(x) is given by 



vT^7 f-i k\ ^ 



c(k) (0) k _^ (2k)l k 

Af— - L^ k] x 2^2 2k [kl] 2X 



(5.104) 

'-'MAM- 
*=0 &=0 



The ratio test [7, p. 709] confirms that this series converges if \x\ < 1. Using 
(5.102) and (5.104), it is possible to show that 

1 oo 

S(x,t) = -== =Vfi,(x)f", (5.105) 

Jl-2xt + t 2 ^ 

so this is the generating function for the Legendre polynomials P n (x). We observe 

that 

dS x — t x — t 

— = — — = -5. (5.106) 

3f [1 -2xf + ? 2 ] 3 / 2 l-2xf + f 2 

Also 

3 S °° 

— = Y,nPnix)t n -\ (5.107) 

n=0 



Equating (5.107) and (5.106), we have 

oo oo 



x — t 

«=0 «=0 

which becomes 



x 

n=0 n=0 n=0 n=0 



Y j P„(x)t n -J^P n (x)t n+1 = ^nPnix)!"- 1 -2xY,nP n (x)t n 

Q 

OO 

J2nPn(x)t n+1 , 



H = 
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and if P_i(x) — 0, then this becomes 



n=0 



n=0 



n=0 



X J2 P n(x)t" ~ J2 P n-l(x)t" = Y, {H + VPn+l(x)t" - 2x^nP n (x)t n 

=0 

oo 



n=0 

oo 



n=0 

so on comparing like powers of t in this expression, we obtain 

xP„(x) - P„-i(x) — (n + l)P n+ i(x) - 2xnP n (x) + (n - \)P n -\(x), 
which finally yields 

(n + l)P n+ i{x) = (2n+ l)xP„(x)-nP n -i(x) 
or (for n > 1) 



2m + 1 n 

P n +l(x) = — -xP n (x) — -P n -i(x), 

n + 1 n + 1 



(5.108) 



which is the three-term recurrence relation for the Legendre polynomials, and hence 
is yet another special case of (5.5). 



x 
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Figure 5.3 Legendre polynomials of degrees 2, 3, 4, 5. 
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The Legendre polynomials arise in potential problems in electromagnetics. Mod- 
eling the scattering of electromagnetic radiation by particles involves working with 
these polynomials. Legendre polynomials appear in quantum mechanics as part 
of the solution to Schrodinger's equation for the hydrogen atom. Some Legendre 
polynomials appear in Fig. 5.3. 

5.6 AN EXAMPLE OF ORTHOGONAL POLYNOMIAL 
LEAST-SQUARES APPROXIMATION 

Chebyshev polynomials of the first kind, and Legendre polynomials are both orthog- 
onal on [— 1, 1], but their weighting functions are different. We shall illustrate the 
approximation behavior of these polynomials through an example wherein we wish 
to approximate 

0, -1 <x < 



/(*) = ( ,; <-<! (5-109) 

by both of these polynomial types. We will work with fifth-degree least-squares 
polynomial approximations in both cases. 

We consider Legendre polynomial approximation first. We must therefore nor- 
malize the polynomials so that our basis functions have unit norm. Thus, our 
approximation will be 

5 

f(x) = J^a k <t> k (x), (5.110) 

k=0 



where 



and 



1 

4>k(x) — Pk(x), 

llfltll 



f 1 1 

= / fix) 

J -I lift II 



ak = {f,<Pk)=l TT7rrJi x> > P kix)dx, 
for which we see that 

' Pk(x) 



hx) = J2 



fc=0 



J fix)P k ix)dx 



\\pk\ 



(5.111) 



We have [e.g., via (5.102)] 



P ix) = 1, Pi(x) = x, P 2 (x) = l -YSx 2 - 1], 

P 3 (x) = -[5x 3 - 3x], P 4 (x) = -[35jc 4 - 30x 2 + 3], 
2 8 

P 5 (x) = -[63x 5 - 70x 3 + 15x]. 
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The squared norms [via (5.98)] are 



||Poll 2 = 2, ||Pil| 2 = -, 

IIAII 2 = ^, ll^ll 2 ^-^-- 



Iftll^j. 



i^ir = -, 



By direct calculation, a# — L P/ C (x)dx becomes 



a 



a 2 



a 3 



a4 



0-5 



/ I ■ dx — l,a\ — J 

Jq Jo 

i /•* 

- / [5x 3 - 3x 
2 Jo 

s/o' 135 ' 

8 Jo 



1 1 

x dx 



2' 



\dx 



1 dx 



il 



Jo 



1 



Mo = 0, 



-ii 



Jo 



30x 2 + 3] dx 



35 



30 



0, 



[63x 5 -70x 3 + 15x]dx 



63 



r 1 fii 


l 


70 


n <i 


1 15 


ri o" 


1 


-X 




- 


-X 


H 


-X 




1.6 





8 


L4 


o « 


L2 






16 



The substitution of these (and the squared norms ||/\ll 2 ) into (5.111) yields the 
least-squares Legendre polynomial approximation 



fix) = ^PoOt) + /l(l) " 7^3« + ^rP 5 (x). 

2 4 16 32 



(5.112) 



We observe that /(0) = 5, /(l) = 52 = 1.15625, and /(-l) = -^ = -0.15625. 
Now we consider the Chebyshev polynomial approximation. In this case 



where 



and 



/(•*) = ^2,h<t>kix), 



(5.113) 



<t>kix) = 



1 



;T k ix), 



b k = {f,4>k) 



1 7*|| 
1 1 fix)T k ix) 



J-iVT=x~ 2 



\T k \ 



dx, 
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from which we see that 



/(*) = £ 



k=0 



J-iVT- 



Vi - .V 



:/(x)TJt(x) dx 



117*11 



r **(*). 



=ft 



We have [e.g., via (5.57)] the polynomials 

T (x) = l, Ti(x)=x, T 2 (x) = 2x 2 - 1, 

r 3 (x) = 4x 3 -3x, r 4 (x) = 8a 4 -8x 2 + 1, 

75(a) = 16x 5 -20x 3 +5x. 

The squared norms [via (5.55)] are given by 



(5.114) 



IITbll -ff,||r k || =- (*>!)• 

By direct calculation /^ = J , T^jx) dx becomes (using x = cos#, and 

y'l—x 2 

T k (cos0) = cos(k9) [recall (5.53)] 0* = Jq /2 cos(k6)d6, and hence 



fo 


= / 1 • d0 = -, 

Jo 2' 








h 


= / cos(9d6» = [sin^Q 7 = 1, 
Jo 




h 


i-JT/2 

= / cos(2(9) J6» = 


- sm(26) 


tt/2 



= 0, 


h 


,71/2 

= / cos(36)d6 = 
Jo 


- sin(36>) 


tt/2 



1 

~~3' 


h 


/.JT/2 

= / cos(4(9) dO = 
Jo 


"1 

- sin (46») 
.4 


tt/2 



= 0, 


h 


rn/2 
= / cos(50)J6» = 
Jo 


"1 
- sin(50) 


tt/2 



1 

~ 5" 



Substituting these (and the squared norms ||7i|| 2 ) into (5.114) yields the least- 
squares Chebyshev polynomial approximation 



fix) = - 



|r (x) +2T l (x) - -r 3 (x) + -7s(x) 



(5.115) 



We observe that /(0) = ±, /(l) = 1.051737, and /(-l) = -0.051737. 
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Figure 5.4 Plots of f(x) from (5.109), the Legendre approximation (5.112), and the 
Chebyshev approximation (5.115). The least-squares approximations are fifth-degree poly- 
nomials in both cases. 



Plots of f(x) from (5.109) and the approximations in (5.112) and (5.115) appear 
in Fig. 5.4. Both polynomial approximations are fifth-degree polynomials, and yet 
they look fairly different. This is so because the weighting functions are different. 
The Chebyshev approximation is better than the Legendre approximation near x — 
±1 because greater weight is given to errors near the ends of the interval [—1, 1]. 

5.7 UNIFORM APPROXIMATION 



The subject of uniform approximation is distinctly more complicated than that of 
least-squares approximation. So, we will not devote too much space to it in this 
book. We will concentrate on a relatively simple illustration of the main idea. 

Recall the space C[a, b] from Chapter 1. We recall that it is a normed space 
with the norm 

||*|| = sup \x(t)\ (5.116) 

t€[a,b] 

for any x(t) e C[a, b]. In fact, further recalling Chapter 3, this space happens to 
be a complete metric space for which the metric induced by (5.116) is 



d(x, y) = sup \x{t) — y{t)\, 

t€[a,b] 



(5.117) 
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where x(t), y(t) e C[a, b]. In uniform approximation we (for example) may wish 
to approximate f(t) e C[a, b] with x(t) e P"[o, b] C C[a,b] such that \\f — x\\ 
is minimized. The norm (5.116) is sometimes called the Chebyshev norm, and so 
our problem is sometimes called the Chebyshev approximation problem. The error 
e(t) — f(t) — x(t) has the norm 

|M|= sup |/(f) -*(0I (5-118) 

t€[a,b] 

and is the maximum deviation between f(t) and x(t) on [a, b]. We wish to find x(t) 
to minimize this. Consequently, uniform approximation is sometimes also called 
minimax approximation as we wish to minimize the maximum deviation between 
f(t) and its approximation x(t). We remark that because f(t) is continuous (by 
definition) it will have a well-defined maximum on [a, b] (although this maximum 
need not be at a unique location). We are therefore at liberty to replace "sup" by 
"max" in (5.116), (5.117) and (5.118) if we wish. 

Suppose that yj (t) e C [a , b] for j = 0, 1, ...,« — 1 and the set of functions 
[yj(t)\j e Z„} are an independent set. This set generates an w-dimensional sub- 
space 4 of C[a,b] that may be denoted by Y — \YH)l=o a iyj( t )\ ol i e R|. From 
Kreyszig [8, p. 337] consider 

Definition 5.1: Haar Condition Subspace Y C C[a, b] satisfies the Haar 
condition if every y e Y (y ^ 0) has at most n — \ zeros in [a, b], where n — 
dim(F) (dimension of the subspace Y). 

We may select y/(0 = ? ; for j e Z„ so any y(t) e Y has the form y(t) — 
S'!=o a ; fJ anc ^ mus ' s a polynomial of degree at most n — 1. A degree n — 1 
polynomial has n — \ zeros, and so such a subspace Y satisfies the Haar condition. 

Definition 5.2: Alternating Set Let x e C[a, b], and y e Y, where Y is any 
subspace of C[a, b]. A set of points to, . . . , t\ in [a, b] such that to < t\ < ■ ■ ■ < tk 
is called an alternating set for x — y if x(tA — y(tj) has alternately the values 
+ ||x — y\\, and — \\x — y\\ at consecutive points tt. 

Thus, suppose x(tj) - y(tj) = +\\x - y\\, then x(t j+i ) - y(t j+i ) = -\\x - y\\, 
but if instead x(t j ) — y(tj) — —\\x — y\\, thenx(f /+ i) — y(tj + \) = +\\x — y\\. The 
norm is, of course, that in (5.116). 

Lemma 5.2: Best Approximation Let Y be any subspace of C[a, b] that 
satisfies the Haar condition. Given f e C[a, b], let y e Y be such that for / — y 
there exists an alternating set of n + 1 points, where n — dim(Y), then y is the 
best uniform approximation of / out of Y. 



The proof is omitted, but may be found in Kreyszig [8, pp. 345-346]. 

A subspace of a vector space X is a nonempty subset Y of X such t 
scalars a, b from the field of the vector space we have ay\ + byx e Y. 



A subspace of a vector space X is a nonempty subset Y of X such that for any y\ , yi e Y , and all 
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Consider the particular case of C[— 1, 1] with f(t) e C[— 1, 1] such that for a 
given « € N 

f(t) = t". (5.119) 

Now Y — Z]/=o a / f/ l Q, 7 e R , so yj(t) — t j for _/' € Z„. We wish to select a,- 
such that the error e — f — y 

n-\ 

e(t) = f(t)-J^ajtJ (5.120) 

7=0 

is minimized with respect to the Chebyshev norm (5.116); that is, select a,- to 
minimize ||e||. Clearly, dim(7) = n. According to Lemma 5.2 ||e|| is minimized 
if e(t) in (5.120) has an alternating set of n + 1 points. 

Recall Lemma 5.1 (Section 5.3), which stated [see (5.44)] that 

n 

cos n6 = J^Pn.k cos k 0. (5.121) 

k=Q 

From (5.46) Pn+i,n+i = 2/J B> „, and /3 ,o = 1. Consequently, /3„,„ = 2"" 1 (for « > 
1). Thus, (5.121) can be rewritten as 

n-\ 

cos«0 = 2" _1 cos" 9 + ^Pn,j cos J ' (5.122) 

.7=0 

(n > 1). Suppose that t — cos0, so 6 e [0, n] maps to f e [— 1, 1] and from (5.122) 

B-l 

cos[n cos -1 f] = 2"- 1 f" + ^/S nJ ^. (5.123) 

j=o 

We observe that cos nQ has an alternating set of n + 1 points 0q , 0i , . . . , 0„ on [0, it ] 
for which cos n9u = ±1 (clearly || costi0|| = 1). For example, if n — 1, then 

6» = 0, $i = it. 

If n = 2, then 

(9 = 0, 01 = 7T/2, 2 = 7T, 

and if n = 3, then 

7T 27T 

00 = 0, 01 = -, 9 2 = — , 03 = JT. 

In general, 0£ = -tt for fc = 0, 1, ... , n. Thus, if f& = cos 0,1 then fj = cos (-?r). 
We may rewrite (5.123) as 

1 1 ""' 

^ cos[« cos" 1 f] = t" + —J2Pnjt J - (5.124) 

7=0 
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This is identical to e{t) in (5.120) if we set fi n j — —2" ay. In other words, if we 
choose 

1 _, 

e(t) = — — j- cos[w cos t], (5.125) 

then ||e|| is minimized since we know that e has an alternating set t — tk — 
cos (f tt), k = 0, 1, . . . , n, andft e [-1, 1]. We recall from Section 5.3 that T k (t) — 
cos[k cos -1 1] is the /cth-degree Chebyshev polynomial of the first kind [see (5.53)]. 
So, e{t) = T n (t)/2 n ~ 1 . Knowing this, we may readily determine the optimal coef- 
ficients at in (5.120) if we so desire. 

Thus, the Chebyshev polynomials of the first kind determine the best degree 
n — 1 polynomial uniform approximation to /(f) = f" on the interval t € [—1, 1]. 
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PROBLEMS 

5.1. Suppose that {4>k(x)\k e Z^} are orthogonal polynomials on the interval D ■■ 
[a, b] C R with respect to some weighting function w(x) > 0. Show that 



N-l 



^ ak<j>k(x) = 



k=0 

holds for all x e [a, b] iff a^ — for all k e Zn [i.e., prove that {<pk{x)\k € 
Z;v} is an independent set]. 

5.2. Verify (5.11) by direct calculation for 4>k(x) — Tk(x)/\\Tk\\ with n — 2; that 
is, find 4>q{x), 4>\{x), <p2(x), and foix), and verify that the left- and right-hand 
sides of (5.11) are equal to each other. 
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5.3. Suppose that [<p n (x)\n e Z + } are orthogonal polynomials on the interval 
[a, b] with respect to some weighting function w(x) > 0. Prove the fol- 
lowing theorem: The roots xj (j = 1, 2, . . . , n) of <j> n (x) — (n € N) are all 
real-valued, simple, and a < xj < b for all j. 

5.4. Recall that 7}(x) is the degree j Chebyshev polynomial of the first kind. 
Recall from Lemma 5.1 that cos(n6) = Y^!k=ofin,k cos^ 6. For x e [— 1, 1] 
with k e Z + , we can find coefficients aj such that 

k 
x — 2_\ajTj(x). 



Prove that 



w^'-l 



a; = K^TPir cos k+r 6d6. 



Is this the best way to find the coefficients a ; ? If not, specify an alternative 
approach. 

5.5. This problem is about the converse to Lemma 5.1. Suppose that we have 
p(cos0) — J]^ = o a 3,*: cosk ®- We wish to find coefficients b^ j such that 

3 

p(cos0) = ^hj cos(j6). 

7=0 

Use Lemma 5.1 to show that a — Ub, where 

a = [03,0 03,1 03,2 03,3]^, b = [/j 3i0 63,1 ^3,2 ^3,3] r 

and U is an upper triangular matrix containing the coefficients p m % for 
m — 0, 1, 2, 3, and fc = 0, 1 . . . , m. Of course, we may use back-substitution 
to solve Ub = a for b if this were desired. 

5.6. Suppose that for n e Z + we are given p(cos9) — Yll =o a n,kcos k 0. Show 
how to find the coefficients b n j such that 



p(cos6) = ^/3„jcos(j0), 
;'=o 

that is, generalize the previous problem from n — 3 to any n. 
5.7. Recall Section 5.6, where 

., , JO, -l<x<0 
/W = 1, 0<*<1 
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was approximated by 

" „ 1 

for n = 5. Find a general expression for b k for all k e Z + . Use the resulting 
expansion 



to prove that 



5.8. Suppose that 



then find b k in 



- = E^— (-1)"- 

4 ^ 2« - 1 

n = \ 



,0, -1 < X < 
/(X) H _ 0<X<1 



oo 1 

/(jc) = y^Sk — T k (x). 



5.9. Do the following: 



(a) Solve the polynomial equation 7^(x) = for all k > 0, and so find all 
the zeros of the polynomials 7i(x). 

(b) Show that T n {x) satisfies the differential equation 

(1 - x 2 )T( 2) (x) - xT^\x) + n 2 T n (x) = 0. 

Recall that t£\x) = d r T n {x)/dx r . 

5.10. The three-term recurrence relation for the Chebyshev polynomials of the sec- 
ond kind is 

U r+ l(x) = 2xU r (x)-U r -l(x), (5.P.1) 

where r > 1, and where the initial conditions are 

U (x) = l,Ui(x) = 2x. (5.P.2) 

We remark that (5.P.1) is identical to (5.57). Specifically, the recursion for 
Chebyshev polynomials of both kinds is the same, except that the initial 
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conditions in (5. P. 2) are not the same for both. [From (5.56) Tq(x) — 1, but 
T\(x) — x.] Since deg(U r ) — r, we have (for r > 0) 



U r {x) = J2 u r,jx j - 

.7=0 



(5.P.3) 



[Recall the notation for cf> r (x) in (5.3).] From the polynomial recursion in 
(5.P.1) we may obtain 



Ur+l,j — 2U,-,j-l — Ur-lJ. 



(5.P.4a) 



This expression holds for r > 1, and for j = 0, 1, . . . , r, r + 1. From (5. P. 2) 
the initial conditions for (5 .P. 4a) are 



I/o,o = 1, U h0 = 0, Ui,i = 2. 



(5.P.4b) 



Write a MATLAB function that uses (5.P.4) to generate the Chebyshev poly- 
nomials of the second kind for r = 0, 1, . . . , N. Test your program out for 
N — 8. Program output should be in the form of a table that is written to a 
file. The tabular format should be something like 



degree 





1 


2 


3 


4 


5 


6 


7 


8 





1 


























1 





2 























2 


-1 





4 




















3 





-4 





8 


















etc. 



5.11. The Chebyshev polynomials of the second kind use 

D = [-1, 1], w(x) = VI -x 2 . 



(5.P.5) 



Denote these polynomials as the set {<p n (x)\n e Z + }. (This notation applies 
if the polynomials are normalized to possess unity-valued norms.) Of course, 
in our function space, we use the inner product 



(f,g)=j Jl-x 2 f(x)g(x)dx. 



(5.P.6) 



Derive the polynomials {<j) n (x)\n e Z + }. (Hint: The process is much the 
same as the derivation of the Chebyshev polynomials of the first kind pre- 
sented in Section 5.3.) Therefore, begin by considering q r -\(x) which is 
any polynomial of degree not more than r — 1. Thus, for suitable c& € R, we 
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must have q r -\{x) = Yfj^CjfyW'Vnd ((j> r ,q r -i) = Ej-=o c ./( < / > r, 4>j) = 
because (</> r ,<pj) — for j — 0, 1, . . . , r — 1. In expanded form 



(4> r ,q r -i)= I v 1 — x 2 4> r (x)q r -\(x) dx — 0. (5. P. 7) 

Use the change of variable x — cos 9 (so Jx = — sin 9 dO) to reduce (5 .P. 7) to 



f 

Jo 



sin 2 6»cos(A:6')(/) r (cos0)d6' = (5.P.8) 



for k = 0, 1, 2, . . . , r — 1, where Lemma 5.1 and its converse have also been 
employed. Next consider the candidate 

sin(r + 1)6* 
r (cose) = C r . - , (5.P.9) 

sm9 

and verify that this satisfies (5. P. 8). [Hence (5. P. 9) satisfies (5.P.7).] Show 
that (5. P. 9) becomes 

sin[(r + 1) cos" 1 x] 

<t> r (x) = C r — 

Prove that for C r = 1 (all r € Z+) 

&r+l(x) = 2x<j) r (x) - ^ r -l(x). 

In this case we normally use the notation C/ r (x) = 4> r (x). Verify that Uq(x) — 
1, and that Ui(x) = 2x. Prove that \\U n \\ 2 = f for n € Z+. Finally, of 
course, </> n (x) = U n (x)/\\U n \\. 

5.12. Write a MATLAB function to produce plots of Uk(x) for k — 2, 3, 4, 5 similar 
to Fig. 5.1. 

5.13. For <pk(x) — Uk(x)/\\Uk\\, we have 

[2 
4>k(x) = J —U k (x). 
V n 

Since (<j>k,<l>j) = &k-j, f° r an Y /(^) € L 2 [— 1, 1], we have the series 
expansion 



f(x) = J^a/i(/)t(x), 



*:=0 

where 



«* = (/, <Pk) = 7777-7 / \/l -x 2 f(x)U k (x)dx. 

\\Uk\\ J-i 
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Suppose that we work with the following function: 

0, -1 <x < 



/(A) " ' 1, 0<x<l 

(a) Find a nice general formula for the elements of the sequence (ai) 

(k € Z+). 

(b) Use MATLAB to plot the approximation 



hi*) = ^2,ak(j)k(x) 



k=Q 



on the same graph as that of f(x) (i.e., create a plot similar to that of 
Fig. 5.4). Suppose that x[fk(x) — 7]t(;c)/||7]t||; then another approxima- 
tion to f(x) is given by 



f\(x) = ^2,b k f k {x). 



fc=0 

Plot fi(x) on the same graph as feix) and f(x). 
(c) Compare the accuracy of the approximations f\{x) and f2(x) to fix) 
near the endpoints x = ±1. Which approximation is better near these 
endpoints ? Explain why if you can. 

5.14. Prove the following: 

(a) T„(x), and T n -\{x) have no zeros in common. 

(b) Between any two neighboring zeros of T n (x), there is precisely one zero 
of T n _\(x). This is called the interleaving of zeros property. 

{Comment: The interleaving of zeros property is possessed by all orthog- 
onal polynomials. When this property is combined with ideas from later 
chapters, it can be used to provide algorithms to find the zeros of orthogonal 
polynomials in general.) 



5.15. 


(a) 


Show that we can write 


















T„(x) = cos[n 


cos 


'*] = 


cosh[ncosh x]. 










[Hint: Note that cosjc = j(e 


Jx _|_ 


e-J x ), 


cosh* = 


\{e x +e~ 


~ x ), so 


that 






cosx = cosh(jx).] 
















(b) Prove that T 2n (x) = T n (2x 2 


-1). 


[Hint: 


cos(2x) : 


- 2 cos 2 x 


-1.] 





5.16. Use Eq. (5.63) in the following problems: 
(a) Show that 



— H r (x) = 2rH r - 1 (x). 

ax 
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(b) Show that 



d 

dx 



'id 

e~ x — H r (x) 

dx 



= -2re~ x2 H r (x). 



(c) From the preceding confirm that H r (x) satisfies the Hermite differential 
equation 

H< 2) (x) - 2xH { r l \x) + 2rH r (x) = 0. 

5.17. Find H k (x) for k — 0, 1, 2, 3, 4, 5 (i.e., find the first six Hermite polynomials) 
using (5.83). 

5.18. Using (5.63) prove that 

N 2 n ~ 2 J 

H k (x)=n\Y(-l) j x"- 2j , 

~ JKn- 2j)\ 

where N — n/2 (n is even), N = (n — l)/2 (« is odd). 

5.19. Suppose that Pk(x) is the Legendre polynomial of degree k. Recall Eq. 
(5.90). Find constants a and fi such that for Qk(x) = Pk(<xx + fi) we have 
(for k ± r) 



L 



b 

Q r (x)Q k (x)dx = 0. 



[Comment: This linear transformation of variable allows us to least-squares 
approximate f(x) using Legendre polynomial series on any interval [a, b].] 

5.20. Recall Eq. (5.105): 

1 oo 

-== = = TP n ( X )t". 

VT^2x7TP ^ Q 

Verify the terms n = 0, 1, 2, 3. 

[Hint: Recall Eq. (3.82) (From Chapter 3).] 

5.21. The distance between two points A and B in R 3 is r, while the distance from 
A to the origin O is r\, and the distance from B to the origin O is ri. The 
angle between the vector OA, and vector OB is 9. 

(a) Show that 

1 1 



\ r i + r 2 ~ 2 nr 2 cos0 
[Hint: Recall the law of cosines (Section 4.6).] 
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(b) Show that 



1 1 ^ n \ n 

r r 2~^ \ r 2/ 



where, of course, P n (x) is the Legendre polynomial of degree n. 
{Comment: This result is important in electromagnetic potential theory.) 

5.22. Recall Section 5.7. Write a MATLAB function to plot on the same graph 
both f(t) = t" and e(t) [in (5.125)] for each of n = 2, 3, 4, 5. You must 
generate four separate plots, one for each instance of n. 

5.23. The set of points C — {e' e \G e R, j — V~ 1} is the unit circle of the complex 
plane. If R(e^ e ) > for all OeR, then we may define an inner product on 
a suitable space of functions that are defined on C 

1 C n 
(F,G} = — / R(e je )F*(e je )G(e je )de, (5.P.10) 

2jt J_„ 

where, in general, F(e-* e ), G(e^ 9 ) e C. Since e^ is 2n -periodic in 0, the 
integration limits in (5. P. 10) are — jt to n , but another standard choice is 
from to lit. The function R(e' e ) is a weighting function for the inner 
product in (5. P. 10). For R(e je ), there will be a real-valued sequence (t>) 
such that 



R(e je )= J2 r * e ~ 



jkt) 



and we also have r_^ = r^ [i.e., (r^) is a symmetric sequence]. For F(e J ) 
and G(eJ s ), we have real-valued sequences (/*), and (gt) such that 

oo oo 

&= — oo fc=— oo 

(a) Show that (e^'" , e"^) = r„_ m . 

(b) Show that ie'-i" 6 F(ei e ), e-'" e G(<?> )) = (F(e' e ), G(e /0 )>. 

(c) Show that {F{e-' e ), G(e je )) = (1, F(e~J e )G(eJ e )). 

[Comment: The unit circle C is of central importance in the theory of stability 
of linear time-invariant (LTI) discrete-time systems, and so appears in the 
subjects of digital control and digital signal processing.] 

5.24. Recall Problem 4.21 from and the previous problem (5.23). Given a n (z) — 
E!t=o a n,kZ~ k (and z = e je ), show that 



(a n (z), Z - k ) = ^S k 



for k = 0, l,...,/i -1. 
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[Comment: This result ultimately leads to an alternative derivation of the 
Levinson-Durbin algorithm, and suggests that this algorithm actually gen- 
erates a sequence of orthogonal polynomials on the unit circle C. These 
polynomials are in the indeterminate z~ l (instead of x).] 

5.25. It is possible to construct orthogonal polynomials on discrete sets. This prob- 
lem is about a particular example of this. Suppose that 



b r [n] = J2' 



where n e [-L, U] C Z, U > L > and <j> r ^- ^ 0, so that deg(^ r ) = r, and 
let us also assume that (f) r j e R for all r, and j. We say that {4> r [n\\n e Z + } 
is an orthogonal set if (4>k[n], 4> m [n]) = \\4>k\\ 2 &k-m, where the inner product 
is defined by 

U 
(f[n],g[n])= J2 wM/MsM. (5.P.11) 



and w[n] > for all n e [-L, U] is a weighting sequence for our inner 
product space. (Of course, ||/|| 2 = (/[«], f[n]).) Suppose that L — U = M; 
then, for w[n] — 1 (all n e [-M, M]), it can be shown (with much effort) 
that the Gram polynomials are given by the three-term recurrence relation 

2(2k +1) k 2M + k + 1 

Pk+\ \n\ — npk\n\ Pk-\ M, 

/+ (k+l)(2M-k)' k+l 2M-k lk[lh 

(5. P. 12) 
where po[n] — 1, and pi[n] = n/M. 

(a) Use (5.P.12) to find p k [n] for k = 2, 3, 4, where M = 2. 

(b) Use (5.P.12) to find p k [n] for fc = 2, 3, 4, where M = 3. 

(c) Use (5.P.11) to find \\p k \\ 2 for *; = 2, 3, 4, where M = 2. 

(d) Use (5.P.11) to find || W || 2 for fc = 2, 3,4, where M = 3. 

(Comment: The uniform weighting function w[m] = 1 makes the Gram poly- 
nomials the discrete version of the Legendre polynomials. The Gram poly- 
nomials were actually invented by Chebyshev.) 

5.26. Integration by parts is clearly quite important in analysis (e.g., recall Section 
3.6). To derive the Gram polynomials (previous problem) makes use of sum- 
mation by parts. 

Suppose that v[n], u[n], and f[n] are defined on Z. Define the forward 
difference operator A according to 

A/M = f[n + 1] - f[n] 
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[for any sequence (/[«])]. Prove the expression for summation by parts, 
which is 



v. 



u 



V' u[n]Av[n] — u[n]v[n] ^J 1 — V^ v[n + l]Au[n] 



n=—L 



n=—L 



[Hint: Show that 

«[n]A«[n] = Au[n]i)[n] — v[n + 1]Am[«], 
and then consider using the identity 

u u 

J2 A/[n] = J2 (/[" + 1] " /W) = f [n] 

n——L n=—L 

Of course, f[n]\ B A = f[B] - f[A].] 



u+i 

-L 
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6.1 INTRODUCTION 

Suppose that we have the data {(tk, x(tk))\k e Z„+i}, perhaps obtained experimen- 
tally. An example of this appeared in Section 4.6 . In this case we assumed that 
t k — kT s for which x(tk) = x(t)\ t =kT s are the samples of some analog signal. In 
this example these time samples were of simulated (and highly oversimplified) 
physiological data for human patients (e.g., blood pressure, heart rate, body core 
temperature). Our problem involved assuming a model for the data; thus, assum- 
ing x(t) is explained by a particular mathematical function with certain unknown 
parameters to be estimated on the basis of the model and the data. In other words, 
we estimate x(t) with x(t, a), where a is the vector of unknown parameters (the 
model parameters to be estimated), and we chose a to minimize the error 

e(t k ) — x(t k ) -x(t k ,a), keZ n+ i (6.1) 

according to some criterion. We have emphasized choosing a to minimize 

n 

V(a) = J2 e2 (tk)- (6-2) 

k=Q 

This was the least-squares approach. However, the idea of choosing a to mini- 
mize max£ € z B+1 \e(t k )\ is an alternative suggested by Section 5.7. Other choices 
are possible. However, no matter what choice we make, in all cases x(t k ,a) is 
not necessarily exactly equal to x(t k ) except perhaps by chance. The problem of 
finding x(t, a) to minimize e(t) in this manner is often called curve fitting. It is to 
be distinguished from interpolation, which may be defined as follows. 

Usually we assume to < t\ < • • • < t n -\ < t n with to — a, and t n — b so that 
t k € [a, b] C R. To interpolate the data {{tk, x(tk))\k e Z n+ i}, we seek a function 
p(t) such that t e [a, b], and 

p(tk) = x{t k ) (6.3) 

for all k e Z„+i. We might know something about the properties of x(t) for t ^ 
tk on interval [a,b], and so we might select p(t) to possess similar properties. 
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However, we emphasize that the interpolating function pit) exactly matches xit) 
at the given sample points t\,k e Z„+i. 

Curve fitting is used when the data are uncertain because of the corrupting effects 
of measurement errors, random noise, or interference. Interpolation is appropriate 
when the data are accurately or exactly known. 

Interpolation is quite important in digital signal processing. For example, ban- 
dlimited signals may need to be interpolated in order to change sampling rates. 
Interpolation is vital in numerical integration methods, as will be seen later. In 
this application the integrand is typically known or can be readily found at some 
finite set of points with significant accuracy. Interpolation at these points leads to 
a function (usually a polynomial) that can be easily integrated, and so provides a 
useful approximation to the given integral. 

This chapter discusses interpolation with polynomials only. In principle it is 
possible to interpolate using other functions (rational functions, trigonometric func- 
tions, etc.) But these other approaches are usually more involved, and so will not 
be considered in this book. 



6.2 LAGRANGE INTERPOLATION 

This chapter presents polynomial interpolation in three different forms. The first 
form, which might be called the direct form, involves obtaining the interpolat- 
ing polynomial by the direct solution of a particular linear system of equations 
(Vandermonde system). The second form is an alternative called Lagrange inter- 
polation and is considered in this section along with the direct form. The third 
form is called Newton interpolation, and is considered in the next section. All 
three approaches give the same polynomial but expressed in different mathemati- 
cal forms each possessing particular advantages and disadvantages. No one form 
is useful in all applications, and this is why we must consider them all. 

As in Section 6.1, we consider the data set {itt, xitk))\k e Z n +i}, and we will 
let Xk = xitk). We wish, as already noted, that our interpolating function be a 
polynomial of degree n: 

n 



For example, if n — 1 ilinear interpolation), then 

1 

.7=0 
1 



Xk+l = YlP^J t i+v 



TLFeBOOK 



LAGRANGE INTERPOLATION 



253 



Pl,0 + P\,\tk = Xk, 
Pl,0 + Pl,lft+1 — Xk+l, 



or in matrix form this becomes 



1 tk 

1 ft+i 



Pi,o 

Phi 



Xk 

Xk+l 



This has a unique solution, provided tk # ft+i because in this instance the matrix 

has determinant det I ' ) = tk+i — tk- Polynomial p\(t) — p\ n + p\ \t 

VI 1 ft+i \) ' 

linearly interpolates the points (tk, Xk), and (tk+i,Xk+i)- As another example, if 

n — 2 (quadratic interpolation), then we have 



Xk 



Xk+l 



Xk+l 



7=0 

2 

7=0 

2 

7=0 



7 



7 

&+2> 



or in matrix form 



ft 

ft+i 



1 ft+2 t 



a -i r 



k+1 

2 



£+2 





P2,0 




Xk 




P2,l 


= 


Xk+l 




P2,2 




Xk+l 



(6.5) 



The matrix in (6.5) has the determinant (tk — ft+i)(ft+i — ft+2)(ft+2 — tk), which 
will not be zero if ft, ft+i and £^+2 are all distinct. The polynomial P2(t) — 
P2,o + Pi,\t + p 2 ,2 {2 interpolates the points (t k ,x k ), (t k +i,Xk+i), and (t k+2 , x k+ 2). 
In general, for arbitrary n we have the linear system 



ft 
ft+i 



1 ft+n-l 
1 ft+n 



M-l 



ft+1 



l k+n 



l k+l 



k+n-l k+n-l 
t n—l t n 



l k+n 



Pnfi 




Xk 


Pn,l 




Xk+l 


) n,n — 1 




X k+n-l 


Pn,n 




Xk+n 



(6.6) 
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Matrix A is called a Vandermonde matrix, and the linear system (6.6) is a Vander- 
monde linear system of equations. The solution to (6.6) (if it exists) gives the direct 
form of the interpolating polynomial stated in (6.4). For convenience we will let 
k — 0. If we let t — to, then 





" 1 


t 


■■ t"' 1 


t 




1 


h 


f n — \ 

' ' h 


t 


A = A(t) = 












1 


tn-l ■ 


t n—\ 

' l n-\ 


t" 

n 




1 


tyl 


t n — l 


t 



(6.7) 



Let D(t) = det(A(f)), and we see that D(t) is a polynomial in the indetermi- 
nate t of degree n. Therefore, D{t) = is an equation with exactly n roots (via 
the fundamental theorem of algebra). But D(tk) — for k — 1, . . . , n since the 
rows of A(t) are dependent for any t = t^. So t\, £2, • • • , t n are the only possible 
roots of D(t) — 0. Therefore, if to, t\, . . . , t n are all distinct, then we must have 
det(A(fo)) ^ 0. Hence A in (6.6) will always possess an inverse if tk+i 7^ tk+j for 

For small values of n (e.g., n — 1, or n = 2), the direct solution of (6.6) can be a 
useful method of polynomial interpolation. However, it is known that Vandermonde 
matrices can be very ill-conditioned even for relatively small n (e.g., n > 10 or so). 
This is particularly likely to happen for the common case of equispaced data, i.e., 
the case where tk — to + hk for k — 0, 1, . . . , n with h > 0, as mentioned in Hill [1, 
p. 233]. Thus, we much prefer to avoid interpolating by the direct solution of (6.6) 
when n ^> 2. Also, direct solution of (6.6) is computationally inefficient, unless one 
contemplates the use of fast algorithms for Vandermonde system solution in Golub 
and Van Loan [2]. These are significantly faster than Gaussian elimination as they 
possess asymptotic time complexities of only 0(n 2 ) versus the 0(n 3 ) complexity 
of Gaussian elimination approaches. 

We remark that so far we have proved the existence of a polynomial of degree n 
that interpolates the data {(tk, Xk)\k e Z n+ i}, provided tk # tj (j ^ k). The poly- 
nomial also happens to be unique, a fact readily apparent from the uniqueness of 
the solution to (6.6) assuming the existence condition is met. So, if we can find 
(by any method) any polynomial of degree < n that interpolates the given data, 
then this is the only possible interpolating polynomial for the data. 

Since we disdain the idea of solving (6.6) directly we seek alternative methods 
to obtain p n (t). In this regard better approach to polynomial interpolation is an 
often Lagrange interpolation, which works as follows. 

Again assume that we wish to interpolate the data set {(tk, Xk)\k € X n+ {\. Sup- 
pose that we possess polynomials (called Lagrange polynomials) Lj (t) with the 
property 

' 0, j^k 
1, j=k = 5 ^- 



Lj(t k ) 



(6.8) 
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Then the interpolating polynomial for the data set is 



255 



Pn(t) = * MO+*iMOH \-x n L n (t) = ^XjLjit). 

.7=0 



We observe that 



(6.9) 



Pn(tk) = ^XjLjjtk) = 'Y^Xj&j-k = X<: 
.7=0 .7=0 



(6.10) 



for A: € Z n+ i, so /?«(/;) in (6.9) does indeed interpolate the data set, and via unique- 
ness, if we were to write p n (t) in the direct form p n (t) — Yl"j=o Pn,jt J , then the 
polynomial coefficients p n j would satisfy (6.6). We may see that for j e Z„+i the 
Lagrange polynomials are given by 



Mo=n^ 



(6.11) 






Equation (6.9) is called the Lagrange form of the interpolating polynomial. 

Example 6.1 Consider data set {(to, xq), (t\, x\), (t 2 , x 2 )} so n — 2. Therefore 

(t-h)(t-t 2 ) 





-h)(t - 

-t )(t- 


-h) 
■tii 


(h- 
(t- 


-t )(h ■ 
-t )(t- 


-fe) 
■h) 


2(0 ~ (ft - 


-foXfc- 


-h) 



It is not difficult to see that pi{t) — xoLo(t) + x\L\(t) + X2L2(t). For example, 
p 2 (to) = xoLo(to) + xiLi(to) +x 2 L 2 (to) = xq. 

Suppose that the data set has the specific values {(0, 1), (1, 2), (2, 3)}, and so 
(6.6) becomes 



1 







P2.Q 




1 


1 


1 1 




P2,l 


— 


2 


1 


2 4 




P2,2 




3 



This has solution p 2 (t) — t + 1 (i.e., p 22 — 0). We see that the interpolating poly- 
nomial p n (t) need not have degree exactly equal to n, but can be of lower degree. 
We also see that 



M0 



(r-l)(f-2) 
(0-l)(0-2) 



1 



1 



-(t - l)(t-2) = -(t z -3t + 2), 
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L l{t) = (t - m - 2) - 
(1 - 0)(1 - 2) 


- -tit - 


- 2) = -it 2 - 


-2f) 


L 2 (0= ( '-° )( '- 1) = 

(2-0)(2-l) 


1 
= -tit - 

2 y 


1 9 
- 1) = -{t 1 - 

' 2 


-t). 


Observe that 









x L (t) +x\Li(t) + x 2 L 2 {t) 

= 1 • -it 1 - 3t + 2) + 2 ■ i-\)it 2 - 2t) + 3 • -it 2 - t) 

^ 2 - 2f+ H + H f+4f "B + G' 2 ) =f+1 ' 

which is p 2 (t). As expected, the Lagrange form of the interpolating polynomial, 
and the solution to the Vandermonde system are the same polynomial. 

We remark that once p n (t) is found, we often wish to evaluate p n (t) for t ^ t\ 
(k e Z„+i). Suppose that p n (t) is known in direct form; then this should be done 
using Horner's rule: 

Pnit) = Pnfi + t[p„,l + t[p„,2 + t[p„,3 H h t[Pn,n-l + tp n ,n\] ' ' •]]]• (6-12) 

For example, if n — 3, then 

Vi{t) = P3,o + f[/?3,i + f[P3,2 + fP3,3]]- (6.13) 

To evaluate this requires only 6 flops (3 floating-point multiplications, and 3 floating- 
point additions). Evaluating piit) — /?3,o + P3,\t + pj,^t 2 + P3,3f 3 directly needs 9 
flops (6 floating-point multiplications and 3 floating-point additions). Thus, Horner's 
rule is more efficient from a computational standpoint. Using Horner's rule may be 
described as evaluating the polynomial from the inside out. 

While Lagrange interpolation represents a simple way to solve (6.6) the form 
of the solution is (6.9), and so requires more effort to evaluate at t ^ t\ than the 
direct form, the latter of which is easily evaluated using Horner's rule as noted in 
the previous paragraph. Another objection to Lagrange interpolation is that if we 
were to add elements to our data set, then the calculations for the current data set 
would have to be discarded, and this would force us to begin again. It is possible to 
overcome this inefficiency using Newton's divided-difference method which leads 
to the Newton form of the interpolating polynomial. This may be seen in Hildebrand 
[3, Chapter 2]. We will consider this methodology in the next section. 

When we evaluate p n it) for t ^ tk, but with t e [a, b] = [to, t n ], then this is 
interpolation. If we wish to evaluate p n it) for t < a or t > b, then this is referred 
to as extrapolation. To do this is highly risky. Indeed, even if we constrain t to 
satisfy t e [a, b], the results can be poor. We illustrate with a famous example of 
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Runge's which is described in Forsythe et al. [4, pp. 69-70]. It is very possible 
that for /(f) € C[a, b], we have 

lim sup |/(f) - p„(t)\ = oo. 
Runge's specific example is for f € [—5, 5] with 

where he showed that for any f satisfying 3.64 < \t\ < 5, then 

lim sup|/(f)- p n (t)\ = oo. 

n— >oo 

This divergence with respect to the Chebyshev norm [recall Section 5.7 (of 
Chapter 5) for the definition of this term] is called Runge's phenomenon. A 
fairly detailed account of Runge's example appears in Isaacson and Keller [5, 
pp. 275-279]. 

We close this section with mention of the approximation error /(f) — p n (t). 
Suppose that /(f) € C n+1 [a, b], and it is understood that f^ n+v> {t) is continuous 
as well; that is, /(f) has continuous derivatives f^\t) fork— 1, 2, ...,« + 1. It 
can be shown that for suitable £ = §(f) 

1 " 

e„(t) = /(f) - Pn(t) = - ■ — / ( " +1) (g) ]"[(' - ft), ( 6 - 14 ) 

(=0 

where f € [a, &]. If we know /^ n+1 ^(f), then clearly (6.14) may yield useful bounds 
on |e„(f)|. Of course, if we know nothing about /(f), then (6.14) is useless. 
Equation (6.14) is useless if /(f) is not sufficiently differentiable. A derivation 
of (6.14) is given by Hildebrand [3, pp. 81-83], but we omit it here. However, it 
will follow from results to be considered in the next section. 



6.3 NEWTON INTERPOLATION 

Define 

x(h)-x(to) XI -XQ 

x[to, h]= = . (6.15) 

h — to h — to 

This is called the first divided difference of x(t) relative to t\ and to- We see that 
x[to, fi] = x[t\, to]- We may linearly interpolate x(t) for t e [to, t{\ according to 

t — to 

x(t) « x(t ) H [x(h) - x(t )] = x(to) + (f - t )x[t , h]. (6.16) 

h — to 
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It is convenient to define po(t) — x(to) and p\(t) — x(to) + (t — to)x[to, t\\. This 
is notation consistent with Section 6.2. In fact, p\ (t) agrees with the solution to 



(6.17) 



as we expect. 

Unless x(t) is truly linear the secant slope x[to, t\] will depend on the abscissas 
to and t\. If x(t) is a second-degree polynomial, then x\t\, t] will itself be a linear 
function of t for a given t\. Consequently, the ratio 



" 1 h ' 


Pl.O 


= 


x Q 



x[tQ, h, f2] 



x[t\, t 2 ] - x[t , tl] 
h — to 



(6.18) 



will be independent of to, t\, and t 2 - [This ratio is the second divided difference of 
x(t) with respect to to, t\, and ?2-] To see that this claim is true consider x(t) — 
«o + ci\t + a 2 t 2 , so then 



x[t\, t] = a\ + a 2 (t + t\) . 



(6.19) 



Therefore 



x[h, t 2 ] - x[to, h] — x[h, t 2 ] - x[t\, to] — a 2 (t 2 - to) 
so that x[tQ, t\, t 2 ] — a 2 . We also note from (6.18) that 



1 



x[to 


h 


ti\ = 


ti 


-to 


x[t\ 


to 


ti\ = 




1 


ti 


-h 



X 2 - X\ X\- XQ 



h — h t\ — to 
xi - XQ xq- xi 



h — to to — ti 



(6.20a) 
(6.20b) 



for which we may rewrite these in symmetric form 

xo xi 



x[to, h, t 2 ] = 



xi 



(to - ti)(t Q - t 2 ) (ti - toKh - t 2 ) (h - to)(t 2 - h) 
= x[ti,t ,t 2 ], (6.21) 

that is, x[to, t\, t 2 ] = x[ti, to, t 2 ]. We see from (6.16) that 

x(t) - x(t Q ) 



t-to 



x[to,h], 



that is 



x[t Q ,t] t*tx[to,h], 



(6.22) 
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and from this we consider the difference 

x[t , t] - x[t , h] — x[to, t] -x[h, t ] — (t - h)x[to, h, t] (6.23) 

via (6.18), and the symmetry property x[to, t\,t]= x[t\, to,t]. Since x(t) is assumed 
to be quadratic, we may replace the approximation of (6.16) with the identity 

x(t) =x(t ) + (t-t )x[t ,t] 

= x(to) + (t-to)x[to,ti] +(t - t Q )(t - h)x[t , h, t] (6.24) 

=P\(t) 

via (6.23). The first equality of (6.24) may be verified by direct calculation using 
x(t) — ciq + a\t + ci2t 2 , and (6.19). We see that if pi(t) approximates x(t), then 
the error involved is [from (6.24)] 

e(t) = x(t) - pi(t) = {t- t )(t - ti)x[t , h, t]. (6.25) 

These results generalize to x(t) a polynomial of higher degree. 

We may recursively define the divided differences of orders 0, I, . . . , k — l,k 
according to 

x[t ] = x(t ) =x , 

x(ti)-x(t ) x\ -x 
x[t ,ti] = 



x[to, h, ti\ 



t\ —to t\— to 

x[t\, t 2 ] - x[tp,ti] 

h — to 



x[h, ...,tt] - x[t , ...,t k -i] 
x[to,..-,t k ]= . (6.26) 

tk — to 

We have established the symmetry x[to, h] = x\t\, to] (case k — 1), and also 
x[to, t\, ?2] = x[h, to, ?2] (case k — 2). For k — 2, symmetry of this kind can also 
be deduced from the symmetric form (6.21). It seems reasonable [from (6.21)] that 
in general 

x[f , ...,tk\ 

xo , x\ t x k 



(to - ti) ■ ■ ■ (t - t k ) (h - t ) ■ ■ ■ (ti - t k ) (t k - t ) ■ ■ ■ (t k - t k -i) 

k 
J"— t xj. (6.27) 
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It is convenient to define the coefficient of xj as 

a) = — r—i (6.28) 

>¥./' 

for y = 0, 1, . . . , k. Thus, x[?o, . . . , fjtl = S/=o a / x ;'- We ma y P rove (6-27) for- 
mally by mathematical induction. We outline the detailed approach as follows. 
Suppose that it is true for k — r; that is, assume that 



± ^_ 

hn r i= o(tj-ti) 



x[t ,...,t r ] = l^-—— —xj, (6.29) 

• n 1 1 i=o \tj H) 

and consider [from definition (6.26)] 

1 

X[t , . ..,t r +\] = {X[t\, ..., t r +\] -X[t , ...,t r ]}. (6.30) 

tr+l — tQ 

For (6.30) 



X[t\, ...,t r + \] = 



(h - t 2 ) ■ ■ ■ (h - t r +l) 

X2 , , X r+ \ 



(t 2 ~ h) ■ ■ ■ (t 2 ~ t r+ l) (tr+l ~ h) ■ ■ ■ (tr+l ~ tr) 

(6.31a) 



and 

xq 



x[t , ...,t r ] 



(t ~ h) ■ ■ ■ (t ~ t r ) 

XI 



(h -t )--- (tl - t r ) (t r -to)--- (t r ~ t r -l) 



(6.31b) 



If we substitute (6.31) into (6.30), we see that, for example, for the terms involving 
only x\ 

1 I x\ 



tr+l ~ to I (h - t 2 )(h ~ f 3 ) ' ' ' (tl ~ U)(h - tr+l) 
XI 



(tl - t Q )(h -t 2 )--- (tl - t r -l)(h - t r ) . 

XI 1 f 1 1 



tr+l — to (tl — t 2 ) ■ ■ ■ (tl — t r ) [ tl — t r +i tl — to 
Xl r + i 

= a^ xi. 



(tl ~ t )(tl -t 2 )--- (tl - t r )(h - tr+l) 

The same holds for all remaining terms in x / for j = 0, 1 , . . . , r + 1 . Hence (6.27) 
is valid by induction for all k > 1. Because of (6.27), the ordering of the arguments 
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in x[to, . . . , t k ] is irrelevant. Consequently, x[to, ■ ■ ■ ,t k ] can be expressed as the 
difference between two divided differences of order k — 1, having any k — 1 of 
their k arguments in common, divided by the difference between those arguments 
that are not in common. For example 

X[h,t2,t3] — X[t0,tl,t2] x[to, t2, h] -X[h,t2,t3] 

x[t , h, t 2 , tj,] = = . 

?3 — to tQ — t\ 

What happens if two arguments of a divided difference become equal? The 
situation is reminiscent of Corollary 5.1 (in Chapter 5). For example, suppose t\ — 
t + e ; then 

x(h) - x(t) x(t + e)-x(t) 
x[t, fi] = 



implying that 



so that 



t\ — t 



x(t + e)-x(t) dx(t) 

x[t, t] = lim = 

6^0 e dt 



dx(t) 

x[t,t]= — — . (6.32) 

dt 

This assumes that x(t) is differentiable. By similar reasoning 

d 

—x[to, ...,t k ,t] = x[t Q , ...,t k ,t, t] (6.33) 

dt 

(assuming that to, ■ ■ ■ , tk are constants). Suppose u\, . . . , u n are differentiable func- 
tions of t, then it turns out that 

d y—^ duj 

— x[to, ...,tk,u\,...,u n ]= } x[t Q , ...,tk,u\, ...,u„,Uj]——. (6.34) 
dt "-^ dt 

./ = ! 

Therefore, if u\ = U2 = • • • = u n = t, then, from (6.34) 

d 

—x[to, ■ . . , tk, t, . . . , t] = nx[to, ■ ■ ■ ,h,t, . . . ,t], (6.35) 

n n+1 

Using (6.33), and (6.35) it may be shown that 

d r 

—x[t ,...,t k ,t] = r\ x[t ,...,t k ,t,...,t]. (6.36) 

r+\ 

Of course, this assumes that x(t) is sufficiently differentiable. 
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Equation (6.24) is just a special case of something more general. We note that 
if x(t) is not a quadratic, then (6.24) is only an approximation for t $ {to, t\, t 2 ], 
and so 

x(t) « x(t ) + (t- to)x[t , h] + (t - t )(t - h)x[t Q , fi, t 2 ] = P2(0. (6-37) 

It is easy to verify by direct evaluation that p 2 (t{) — x{t{) for i e {0, 1,2}. Equation 
(6.37) is the second-degree interpolation formula, while (6.16) is the first-degree 
interpolation formula. We may generalize (6.24) [and hence (6.37)] to higher 
degrees by using (6.26); that is 

x(t) = x[t ] + (t- t Q )x[tQ, t], 
x[to, t] = x[to, t\] + (t — h)x[to, t\, t], 

x[to, h,t] = x[tQ,t\,t2] + (t - t2)x[tQ, t\, t2, t], 



x[t , ...,t n -i,t] = x[t , ...,t n ] + (t - t„)x[t Q , ...,t n ,t], (6.38) 

where the last equation follows from 

x[t\, ...,t n ,t] -x[to, ...,t n ] 



x[t , ...,t n ,t] = 



t-t 

x[t , ...,f„-l,f] -x[t , ...,t n ] 
t-tn 



(6.39) 



(the second equality follows by exchanging to and t n ). If the second relation of 
(6.38) is substituted into the first, we obtain 

x(t) = x[t ] + (t- to)x[t , h] + (t- t )(t - ti)x[t , h,t], (6.40) 

which is just (6.24) again. If we substitute the third relation of (6.38) into (6.40), 
we obtain 

x(t) = x[t ] + (t- to)x[to, h] + (t - t )(t - ti)x[to, h,t 2 ] 

+ (t - t )(t - h){t - t 2 )x[t , t\, t 2 , t], (6.41) 

This leads to the third-degree interpolation formula 

p 3 (t) = x[t ] + (f - t Q )x[t , fi] + (t - t )(t - h)x[t , h, t 2 ] 
+ (t - t )(t - h)(t - t 2 )x[t , ti, t 2 , t 3 ]. 
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Continuing in this fashion, we obtain 

x(t) = x[t ] + (t- foMfo, h] + (t- t )(t - h)x[to, h, h] 

+ ••• + (?- t )(t - h) ■ ■ ■ (t - t„-i)x[to, h...,t n ] + e(t), (6.42a) 

where 

e(t) = (t- t Q )(t - h) ■ ■ ■ (t - t n )x[t , t u ...,t n ,t], (6.42b) 

and we define 

Pn(t) = X[t ] + (t - t Q )x[t , fi] 

+ (t- t )(t - h)x[t Q , h,t 2 ] + --- + (t-to)---(t- t„-i)x[to, • ■ • , t n ], 

(6.42c) 
which is the nth-degree interpolating formula, and is clearly a polynomial of degree 
n. So e(t) is the error involved in interpolating x(t) using polynomial p n (t)- It is 
the case that p n (tk) — x(tk) for k = 0, 1, . . . , n. Equation (6.42a) is the Newton 
interpolating formula with divided differences. If x(t) is a polynomial of degree n 
(or less), then e(t) = (all t). This is more formally justified later. 

Example 6.2 Consider e(t) for n = 2. This requires [via (6.27)] 

x x\ 



X[tt), h, t 2 , t] 



(to - h)(fo - t 2 )(to - t) (h - fo)Oi - t 2 )(h - t) 
x 2 x(t) 



(h - t )(t 2 - h){t 2 -t) (t- t )(t - h)(t - t 2 ) 
Thus 



e(t) = (t - t )(t - ti)(t - t 2 )x[t , t u t 2 , t] 

(t - h)(t - t 2 )x _ (t - t )(t - t 2 ) Xl _ (t - t )(t - t 1 )x 2 
(to - h)(t - t 2 ) (h - to)(h - t 2 ) (t 2 -to)(t 2 -h) 



+x(t). 



=-PiV) 

(6.43) 
The form of p 2 (t) in (6.43) is that of p 2 (t) in Example 6.1: 

p 2 (t) = xoL (t) + x\L\(t) + x 2 L 2 (t). 

If x(t) — «o + a\t + a 2 t 2 , then clearly e(t) — for all t. In fact, p 2 t — ak- 

Example 6.3 Suppose that we wish to interpolate x(t) — e* given that t\ — kh 
with h = 0.1, and k = 0, 1, 2, 3. We are told that 

x Q = 1.000000, xi = 1.105171, x 2 = 1.221403, x 3 = 1.349859. 
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= 


*Oo) = 


1.000000 


x[t ,ti] = 


: 1.05171 


= 0.1 


x(h) = 


1.105171 


x[h,t 2 ] = 


: 1.16232 


= 0.2 


x(t 2 ) = 


1.221403 


x[t 2 ,t 3 ] = 


1.28456 


= 0.3 


x(t 3 ) = 


1.349859 
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We will consider n — 2 (i.e., quadratic interpolation). The task is aided if we 
construct the divided- difference table: 



x[t ,ti,t 2 ] = 0.55305 
x[t l ,t 2 ,t 3 ] = 0.61120 



For t € [0, 0.2] we consider [from (6.37)] 

x(t) % x (t ) + (t- t )x[t , ti] + (t- t )(t - h)x[t , ti,t 2 ], 

which for the data we are given becomes 

x(t) % 1.000000+ 1.051710? + 0.55305f(f- 0.1) = p$(t). (6.44a) 

Iff = 0.11, then ^(0.11)= 1.116296, while it turns out that x (0.11) = 1.116278, 
so e(0.11) = jc(O.II) - p^(0.11) = -0.000018. For t € [0.1, 0.3] we might con- 
sider 

x(t) % X (h) + (t- h)x[h, t 2 ] + (t- ti)(t - t 2 )x[h,t 2 , t 3 ], 

which for the given data becomes 

x(t) % 1.105171+ 1.16232(f-0.1) + 0.61120(f-0.1)(f-0.2) = p\(t). 

(6.44b) 
We observe that to calculate p\(t) does not require discarding all the results needed 
to determine pf (£)• We do not need to begin again as with Lagrangian interpolation 
since, for example 

x[h,t 2 ] -x[t ,ti] x[t 2 , ?3] -x[h,t 2 ] 
x[to,h,t 2 \= , x[h,t 2 , ?3J = , 

h — to ?3 — t\ 

and both of these divided differences require x\t\, t 2 ]. If we wanted to use cubic 
interpolation then the table is very easily augmented to include x[tQ, ti, t 2 ,t 3 ]. 
Thus, updating the interpolating polynomial due to the addition of more data to the 
table, or of increasing n, may proceed more efficiently than if we were to employ 
Lagrange interpolation. 

We also observe that both p^it) and p\(t) may be used to interpolate x(t) for 
t e [0.1,0.2]. Which polynomial should be chosen? Ideally, we would select the 
one for which e(t) is the smallest. Practically, this means seeking bounds for \e(f)\ 
and choosing the interpolating polynomial with the best error bound. 
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We have shown that if x(t) is approximated by p n (t), then the error has the form 

e(t) — jt(t)x[t Q , . . .,t n ,t] (6.45a) 

[recall (6.42b)], where 

7t(t) = (t- t )(t - h) ■ ■ ■ (t - t n ), (6.45b) 

which is an degree n + 1 polynomial. The form of the error in (6.45a) can be useful 
in analyzing the accuracy of numerical integration and numerical differentiation 
procedures, but another form of the error can be found. 

We note that e(t) — x(t) - p n (t) [recall (6.42)], so both x(t) - p n (t) and n(t) 
vanish at to, t\, . . . , t n . Consider the linear combination 

X(t) =x(t)~ Pn (t)- KTT(t). (6.46) 

We wish to select k so that X(t) = 0, where 1 ^ tk for k e Z n+ \. Such a k 
exists because n(t) vanishes only at to, h, . . . , t n . Let a — minffo, . . . , t n , 7], b — 
max{?o, • ■ • , t n , J], and define the interval / = [a, b]. By construction, X(t) van- 
ishes at least n + 2 times on /. Rolle's theorem from calculus states that X^it) — 
d k X(t)/dt k vanishes at least n + 2 — k times inside /. Specifically, X^ I+1 ^(f) van- 
ishes at least once inside /. Let this point be called f . Therefore, from (6.46), we 
obtain 

*(»+!>(!) - p^ +x \f) - KJt {n+l \f) = 0. (6.47) 

But p n (t) is a polynomial of degree n, so p^ n+l \t) = for all t e I. From (6.45b) 
jt (n+l) (t) = (n+ 1)!, so finally (6.47) reduces to 

' x (n+1) (f). (6.48) 



(n+l)\ 
From X(i) — in (6.46), and using (6.48), we find that 

e(t) = x(t) - Pn (t) = - x (n+V) (f)7T(t) (6.49) 

(n + 1)! 

for some | e /. If we were to let 1 — tk for any k e Z n+ i, then both sides of (6.49) 
vanish even in this previously excluded case. This allows us to write 

e ^ = / * , * ( " +1) (g(0M0 (6.50) 

(n + 1)! 

for some ^ = f (t) el (I — [a, b] with a — minffo, ■ • ■ , t n , t], and b — 
max{?o, ■ ■ ■ ,t n , t}). If x < - n+1 \t) is continuous for t e I, then x^" +1 '(f) is bounded 
on /, so there is an M n+ \ > such that 

*(»+!>($) < M n+l , (6.51) 
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and hence 



k(OI < , M " + L \*(t)\ (6-52) 

(« + l)! 



for all t e I. It is to be emphasized that for this to hold x ( - n+i Ht) must exist, 
and we normally require it to be continuous, too. Equations (6.50) and (6.45a) are 
equivalent, and thus 

7t(t)x[t , ..., t n , t] = — — — x ( " +1) (^)7r(r), 
(n + 1)! 

or in other words 

x[t , . . . , t n , t] = * — * ( " +1) (l) (6.53) 

(« + 1)! 

for some f € /, whenever x'" +1 ^(f) exists in /. In particular, if jt (?) is a polynomial 
of degree n or less, then (6.53) yields ;c[fo, ...?«, t] = 0, hence e(f) = for all t 
(a fact mentioned earlier). 

Example 6.4 Recall Example 6.3. We considered x(t) = e' with fg = 0, fi = 
0.1, ?2 = 0.2, and we found that 

pf(0= 1.000000+ 1.051710? + 0.553050f(f- 0.1) 

for which pf (0.11) = 1.116296, and x(0.11) = 1.116278. The exact error is 

e(0.11) = x(0.11) - p$(0.U) = -0.000018. 

We will compare this with the bound we obtain from (6.52). Since n = 2, x^(f) = 
e', and / = [0,0.2], so 

M 3 = e 02 = 1.221403 

(which was given data), and n(t) = f(f -0.1)0 - 0.2), so |tt(0.11)| =9.9 x 10" 5 . 
Consequently, from (6.52), we have 

1.221403 . 

|e(0.11)| < j 9.9 x 10" 5 = 0.000020. 

The actual error certainly agrees with this bound. 

We end this section by observing that (6.14) immediately follows from (6.50). 

6.4 HERMITE INTERPOLATION 

In the previous sections polynomial interpolation methods matched the polynomial 
only to the value of the function f(x) at various points x — Xk € [a, b] C R. In 
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this section we consider Hermite interpolation where the interpolating polynomial 
also matches the first derivatives f^\x) at x — x k . This interpolation technique 
is important in the development of higher order numerical integration methods as 
will be seen in Chapter 9. 

The following theorem is the main result, and is essentially Theorem 3.9 from 
Burden and Faires [6]. 

Theorem 6.1: Hermite Interpolation Suppose that f(x) e C l [a, b], and 
that xq,x\, . . . ,x n e [a, b] are distinct, then the unique polynomial of degree (at 
most) In + 1 denoted by p2 n +i(x), and such that 

Pln + 1 [Xj ) = / (Xj ) , p% + , (Xj ) = f m (Xj ) (6.54) 

(J e Z„+i) is given by 

n n 

P2„+i(x) = ^2h k (x)f(x k ) + J^h k (x)f m (x k ), (6.55) 



where 



and 



t=0 k=0 



h k (x) = [1 - 2L ( l\x k ){x - x k )][L k (x)f, (6.56) 



h k (x) = (x - x k )[L k (x)] 2 (6.57) 



such that [recall (6.11)] 

n 

L k (x) = f] ±^±L. (6.58) 

. ^ X k %i 

1=0 

ijtk 

Proof To show (6.54) for xq, x\, . . . , x„, we require that h k (x), and h k (x) in 
(6.56) and (6.57) satisfy the conditions 

h k ( X j) = Sj- k , h ( k 1 \x j ) = (6.59a) 



and 



h k ( Xj ) = 0, h[ l \xj) = t>i- k . (6.59b) 



Assuming that these conditions hold, we may confirm (6.54) as follows. Via (6.55) 

n n 

P2n + l(Xj) = ^h k (xj)f(x k ) + ^2h k (xj)f (1) (x k ), 



k=0 k =0 
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and via (6.59), this becomes 

n n 

k=Q k=0 

This confirms the first case in (6.54). Similarly, via (6.59) 

n n 

*=0 /t=0 

becomes 

P&+i(*j) = E°- /(**) + £*;-*/ (1) (**) = f m (xj), 

k=0 k=0 

which confirms the second case in (6.54). 

Now we will confirm that hk(x), and h k (x) as defined in (6.56), and (6.57) 
satisfy the requirements given in (6.59). The conditions in (6.59b) imply that hk(x) 
must have a double root at x = Xj for j ^ k, and a single root at x — Xk ■ A 
polynomial of degree at most 2n + 1 that satisfies these requirements, and such 
that h^\x k ) — 1 is 

f , , , , (x -x ) 2 ---(x -Xk-i) 2 - (x - x k+i ) 2 ■ ■ ■ (x - x n ) 2 

h k (x) = (x- x k )- 



(x k - x ) z ■■■(x k - x k -\) z ■ (x k - x k+ \Y ■■■{xk- x„) 2 



= (x - x k )L 2 k {x). 

Certainly h k (x k ) — 0. Moreover, h k (x) — L^(x) + 2(x — x k )L k (x)L k (x) so 
h^\xk) = L 2 (x k ) = 1 [via (6.8)]. These verify (6.59b). 

Now we consider (6.59a). These imply xj for j ^ k is a double root of h k (x), 
and we may consider (for suitable a and £> to be found below) 

hk(x) = ™ — r~(x - x ) 2 ■ ■ ■ (x - x k -i) 2 (x - x k+ if 

1 L=oW ~~ x k ) 

i^k 

■ ■ ■ (x — x n ) (ax + b) 
which has degree at most In + 1. More concisely, this polynomial is 



h k (x) = L k (x)(ax + b). 



From (6.59a) we require 



1 = h k (x k ) — L 2 k {x k )(ax k + b) — ax k + b, (6.60) 
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Also, h, (x) = aL k (x) + 2L k (x)L k (x)(ax + b), and we also need [via (6.59a)] 

h k (xk) — ah\{x k ) + 2L k (x k )L k (x k )(ax k + b) = 0, 
but again since L k (x k ) — 1, this expression reduces to 

a + 2L ( k l \x k ) = 0, 
where we have used (6.60). Hence 

a = -2L ( k l) (x k ), b = 1 + 2L ( k 1 \x k )x k . 



Therefore, we finally have 



h k (x) = [1 - 2L ( k l \x k )(x - x k )]L 2 k (x) 



which is (6.56). Since L k {xj) = for j ^ k it is clear that h k {xj) — for j ^ k. 
It is also easy to see that h, (xj) = for all j ^ k too. Thus, (6.59a) is confirmed 
for h k (x) as defined in (6.56). 

An error bound for Hermite interpolation is provided by the expression 

1 " 

/(*) = P2n+lM + .. ; — 11^ " ^) 2 / (2 " +2) «) (6-61) 

(2m + 2)! * = * 

for some f € (a, b), where f(x) e C 2n+2 [a, b]. We shall not derive (6.61) except 
to note that the approach is similar to the derivation of (6.14). Equation (6.14) was 
really derived in Section 6.3. 

In its present form Hermite interpolation requires working with Lagrange poly- 
nomials, and their derivatives. As noted by Burden and Faires [6], this is rather 
tedious (i.e., not computationally efficient). A procedure involving Newton inter- 
polation (recall Section 6.3) may be employed to reduce the labor that would 
otherwise be involved in Hermite interpolation. We do not consider this approach, 
but instead refer the reader to Burden and Faires [6] for the details. We use Hermite 
interpolation in Chapter 9 to develop numerical integration methods, and efficient 
Hermite interpolation is not needed for this purpose. 

6.5 SPLINE INTERPOLATION 

Spline (spliced line) interpolation is a particular kind of piecewise polynomial 
interpolation. We may wish, for example, to approximate f(x) for x € [a, b] C 
R when given the sample points {(x k , f(x k ))\k e Z n+ \] by fitting straight-line 
segments in between (x k , f(x k )), and {x k +\, f(x k+ \)) for k = 0, 1, . . . , n — 1. An 
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^ 




Figure 6.1 The cubic polynomial f{x) = (x — \)(x — 2)(x — 3), and its piecewise linear 
interpolant (dashed line) at the nodes x\ = xq + hk for which xq = 0, and h = A, where 
k = 0, 1 n with n — 8. 



example of this appears in Fig. 6.1. This has a number of disadvantages. Although 
f(x) may be differentiable at x = Xk, the piecewise linear approximation will not 
be (in general). Also, the graph of the interpolant has visually displeasing "kinks" 
in it. If interpolation is for a computer graphics application, or to define the physical 
surface of an automobile body or airplane, then such kinks are seldom acceptable. 
Splines are a means to deal with this problem. It is also worth noting that, more 
recently, splines have found a role in the design of wavelet functions [7, 8], which 
were briefly mentioned in Chapter 1. 

The following definition is taken from Epperson [9], and our exposition of spline 
functions in this section follows that in [9] fairly closely. As always f^(x) — 
d l f(x)/dx l (i.e., this is the notation for the ith derivative of fix)). 

Definition 6.1: Spline Suppose that we are given {{Xk, f(xk))\k e Z n+ \}. 
The piecewise polynomial function p m (x) is called a spline if 



(51) p m (xk) — f(xk) for all k e Z„+i (interpolation). 

(52) lirn 



Pm \x) = lim x 



Pm (x) for all i e Zjv+i (smoothness). 
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(S3) p m (x) is a polynomial of degree no larger than m on every subinterval 
[xk, Xk+\] for k e Z„ (interval of definition). 

We say that m is the degree of approximation and N is the degree of smoothness 
of the spline p m (x). 

There is a relationship between m and N. As there are n subintervals [xjc, X/t+i], 
and each of these is the domain of definition of a degree m polynomial, we see 
that there are Df — n(m + 1) degrees of freedom. Each polynomial is specified by 
m + 1 coefficients, and there are n of these polynomials; hence Df is the number 
of parameters to solve for in total. From Definition 6.1 there are n + 1 interpo- 
lation conditions [axiom (SI)]. And there are n — 1 junction points x\, . . .x n -\ 
(sometimes also called knots), with N + 1 continuity conditions being imposed on 
each of them [axiom (S2)]. As a result, there are D c — («+ 1) + (n — l)(N + 1) 
constraints. Consider 

D f - D c = n(m + 1) - [(« + 1) + (« - l)(N + 1)] =n(m-N-l) + N. 

(6.62) 
It is a common practice to enforce the condition m — N — 1 = 0; that is, we let 

m = N+l. (6.63) 

This relates the degree of approximation to the degree of smoothness in a simple 
manner. Below we will focus our attention exclusively on the special case of the 
cubic splines for which m — 3. From (6.63) we must therefore have N = 2. With 
condition (6.63), then, from (6.62) we have 

D f -D C = N. (6.64) 

As a result, it is necessary to impose TV further constraints on the design problem. 
How this is done is considered in detail below. Since we will look only at m — 3 
with N — 2, we must impose two additional constraints. This will be done by 
imposing one constraint at each endpoint of the interval [a, b]. There is more than 
one way to do this as will be seen later. 

From Definition 6. 1 it superficially appears that we need to compute n different 
polynomials. However, it is possible to recast our problem in terms of B-splines. 
A B-spline acts as a prototype in the formation of a basis set of splines. 

Aside from the assumption that m — 3, N — 2, let us further assume that 

a — xq < x\ < • • • < x n -\ < x n — b (6.65) 

with Xk+i — Xk — h for k = 0, 1, . . . , n — 1. This is the uniform grid assumption. 
We will also need to account for boundary conditions, and this requires us to 
introduce the additional grid points 

x_3 — a — 3h, x-2 — a — 2h, x_\ — a — h (6.66) 



TLFeBOOK 



272 INTERPOLATION 

and 

x n+ 3 = b + 3h, x n+ 2 = b + 2h, x n +\ = b + h. 

Our prototype cubic B-spline will be the function 



(6.67) 



S(x) 



0, 

(x + 2)\ 

1 + 3(x + 1) + 3(x + l) 2 - 3(x + l) 3 , 

1 + 3(1 - x) + 3(1 - xf - 3(1 - x) 3 , 

(2-x) 3 , 

0, 



x < -2 
-2 < x < -1 
-1 <x < 
0<x < 1 

1 < x < 2 
x>2 



(6.68) 



This function has nodes at x e {—2, —1,0, 1 , 2}. A plot of it appears in Fig. 6.2, and 
we see that it has a bell shape similar to the Gaussian pulse we saw in Chapter 3. 
We may verify that S(x) satisfies Definition 6.1 as follows. Plainly, it is piecewise 
cubic (i.e., m — 3), so axiom (S3) holds. The first and second derivatives are, 




Figure 6.2 A plot of the cubic B-spline defined in Eq. (6.68). 
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s (1 \ X ) 



0, 


x < -2 


3(x + 2) 2 , 


-2 < x < -1 


3 + 6(x + l)-9(x+l) 2 , 


-1 < x < 


-3-6(1 — x) + 9(1 -x) 2 , 


<x < 1 


-3(2- x) 2 , 


1 < x < 2 


0, 


x > 2 



(6.69) 



and 



S (2) (x) = 



0, x < -2 

6(x + 2), -2<x<-l 

6-18(x + l), -l<x<0 

6- 18(1 -x), <x < 1 



6(2 -x), 
0. 



1 < x < 2 
x > 2 



(6.70) 



We note that from (6.68)-(6.70) that 



5(0) = 4, 


S(±l)=l, 


5(±2) = 0, 


(6.71a) 


S (1) (0) = 0, 


S (1) (±1) = T 3, 


S (1) (±2) = 0, 


(6.71b) 


S (2) (0) = -12, 


S (2) (±l) = 6, 


S (2) (±2) = 0. 


(6.71c) 



So it is apparent that for i = 0, 1, 2 we have 



?(«')/ 



?(<), 



lim S u; (x) = lim S^(x) 



for all Xk € {—2, —1,0, 1,2}. Thus, the smoothness axiom (S2) is met for N — 2. 
Now we need to consider how we may employ S(x) to approximate any f(x) 
for x e [a, b] C R when working with the grid specified in (6.65)-(6.67). To this 
end we define 

(X — X; \ 
— JT~) (6 ' 72) 



for i = —1,0, l,...,/i,n + l. Since 5 ( u, (x) = ^5 (1) (^), 5, u, (x) = 
l 5 (2) (£^£l) from (6 71 ) ; we have (e g ^ x . ±1 = x . ± jg 



S,-(jc i -) = S(0) = 4, S ( te±i) = S(±l) = l, S I -(x I -± 2 ) = S(±2) = 0, 

(6.73a) 
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S?\ Xi ) = 0, S ; (1) (* (±1 ) = tJ, 5, (1) (x !±2 ) = 0, (6.73b) 

S i 2) ^ = ~^' SfWi) = ^. 5, (2) (x i±2 ) = 0. (6.73c) 

We construct a cubic B-spline interpolant for any f(x) by defining spline pi(x) 
to be a linear combination of Si (x) for i — —1,0, 1, . . . ,n,n + 1, i.e., for suitable 

coefficients a, we have 

«+i 

p 3 (x)= J2 a iS,(x). (6.74) 

The series coefficients a, are determined in order to satisfy axiom (SI) of Definition 
6.1; thus, we have 

n + \ 

f(xk) = ^2 aiSi{x k ) (6.75) 

for & e Z„+i. 

If we apply (6.73a) to (6.75), we observe that 

n+\ 



f(xo) = ^ ajSjixo) = a-iS-i(x Q ) + a Q SQ(xo) + aiSi(xo) 

«+i 
/(*i) = £^ a,-S,-(*i) = a 5 , o(xi)+ai5 , i(x 1 ) + fl 2 52(xi) 



i=-l 



n+1 
f(x n ) — 2_^ a iSi( x n) — Cn-\S n -i(x n ) + a n S n (x n ) + a n+ iS n +\(x n ) 

(6.76) 
For example, for /(xq) in (6.76), we note that Sk(xo) — for k > 2 since (x k — 
xq + A;/;) 

S k (x ) = 5 (^) = S ( *> -»*+*"> ) = S(-*) = 0. 

More generally 

f(x k ) — a k -iS k -i(xk) + a k S k (x k ) + a k+ \S k+ \(x k ) (6.77) 

for which k e X n+ \. Again via (6.73a) we see that 

c , , c f (x + kh) - (xq + (k - l)h) \ 

S k -i(x k ) = S = 5(1) = 1, 
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Sk(xk) = s ■ {X0 + kh)-( X0 + khr = m = 4> 



5/t+l C^yt) = S 



h 



(xo + kh)- (x + (k + \)h) 



S(-l) = l. 



Thus, (6.77) becomes 



«£_! + 4a* + Ojfc+1 = /(**) 



(6.78) 



again for fc = 0, 1, . . . , n. In matrix form we have 



1 


4 


1 


• 


• 











1 


4 


1 • 


• 

















• 


• 4 


1 





_ 








• 


• 1 


4 


1 



a_i 

«o 

a„ 

a n + \ 



=A 





/(*o) 




f(xi) 




f(Xn-l) 




f(x„) 



(6.79) 



We note that A e r("+ 1 ) x ("+ 3 ) ; so there are n + 1 equations in n + 3 unknowns 
(a e R" +3 ). The tridiagonal linear system in (6.79) cannot be solved in its present 
form as we need two additional constraints. This is to be expected from our earlier 
discussion surrounding (6.62)-(6.64). Recall that we have chosen m — 3, so N — 2, 
and so, via (6.64), we have D f — D c — N — 2, implying the need for two more 
constraints. There are two common approaches to obtaining these constraints: 



,(2), 



1. We enforce p^ (xq) 

2. We enforce pf\xo) = / (1) 
or clamped spline). 



pf\x n ) 



(natural spline). 



,(l) 



(xo), and p\ (x n ) = f ( '(x n ) (complete spline 



Of these two choices, the natural spline is a bit easier to work with but can lead 
to larger approximation errors near the interval endpoints xq, and x n compared to 
working with the complete spline. The complete spline can avoid the apparent need 
to know the derivatives f^ l \xo), and f^\x n ) by using numerical approximations 
to the derivative (see Section 9.6). 

We will first consider the case of the natural spline. From (6.74), we obtain 

P3(xq) = a-iS-i(x ) + a So(x ) + aiS\(x ), 



so that 



/?3 2) (*o) = a- 



iS^(*o)- 



a sf?\x ) 



aiS[ 2) (x ). 



(6.80) 
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Since S, (2) (x ) = ±S {2) ( 5fi ^ L ), we have 



l2) _ J_ (2) / x -(x -h) \ _ 1 (2) _ 6_ 



Sri(xo) = —S w ( - — ^ ) = —5^(1) = — , (6.81a) 



^"--X 2 ^) -?*"«»-£ 



S^'(xo) = —S w ( -^-^ I = —S { '(0) = - — , (6.81b) 



,(2)^_, _ 1 eg) { xo ~ (*o + h)\ _ 1 (2) 



where (6.71c) was used. Thus, (6.80) reduces to 

h~ [6a_i — 12«o + 6«i] = 



a_i = 2ao — «i. (6.82) 

Similarly 

Pi\ x n) == Qn—l^n—l\X n ) ~r a n^n\ x n) T fl n + Wn+l (^-n)> 

SO that 

P3 2) (^«) = a«-i5'® 1 (x n ) +a„^ 2) (^«) +« n +i^+ 1 (x„). (6.83) 

Since S, (2) (x„) = ^5 (2) (^^), we have 

c (2) , . _ 1 „( 2 ) f (xQ + nh)-(xQ + (n- l)h) \ _ 1 (2) _ 6 

(6.84a) 



„i 1 m / (xn + m/j) — (xo + n/0\ 1 ,.,■, 12 

&<*,) = 7^5 (2) (^ L_li dj = _ 5 (2) (0 ) = __ (6.84b) 

c (2) , . _ 1 c (2) ( (XQ + w/i ) ~ (*0 + (W+1W \ 1 „( 2 ), n _ 6 



(6.84c) 



where (6.71c) was again employed. Thus, (6.83) reduces to 
h~ [6a„_i — 12a„ + 6a n+ \] — 



o„+i = 2a„ - a„_i. (6.85) 

Now since [from (6.79)] 

a_i + 4a + «i = f(xo) 
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using (6.82) we have 

and similarly 

so via (6.85) we have 



6a = /Oo), 
a„_i + 4a„ + a„+\ = /(*„), 



(6.86a) 



6a„ = /(*„). 
Using (6.86) we may rewrite (6.79) as the linear system 



(6.86b) 



4 


1 • 


• 








1 


4 1 • 


• 











• 


• 1 


4 


1 


_ 


• 


• 


1 


4 _ 



a n -2 
a n -\ 



f(xi) - g/(*o) 

f(X 2 ) 

f(Xn-2) 
_ f(x„-i) - hf(x n ) 



(6.87) 



=/ 

Here we have A e r(" _1 ) x (" -1 ^ so now the tridiagonal system Aa — f has a 
unique solution, assuming that A -1 exists. The existence of A -1 will be justified 
below. 

Now we will consider the case of the complete spline. In the case of 



,(!)/ 



?(!)/ 



?(!)/ 



:■(!)/ 



P^Oo) = a-i^^xo) + ooSq^Cso) + aiS^-'^o) = / (1) (*o)> 
since 5 ( (1) (xo) = ^5 (1) ( a ^ £L ), we have, using (6.71b) 



(6. 



S^(xo) = f 5« ( ^"-^-^ = I 5 (i)(l) = 



r(l)/ 



n 

so (6.88) becomes 



1 -(1) / Xqj-JCo 

h~ V h 
1 



= -S (1) (0) = 0, 
h 



xo - (xq + h) 



,(i) 



is«(-D = -. 
h h 



pl l '(x ) = SA-^-a-i +oi] = / UJ (x ). 



(6.89) 



Similarly 



P \ l) (x„) = a„_i^!_\(x„) + flnS^Oc) + a^i^^fe) = f m (x n ) (6.90) 



TLFeBOOK 



278 INTERPOLATION 

reduces to 

p 3 (x„) = 3h~ 1 [-a„-i +a„ +i ] = / (1) (x„). 

From (6.89) and (6.91), we obtain 

0-1 = 01 - ±fr/ (1) (*o), Cl n +\ = On-l + ^/ (1) (^n)- 

If we substitute (6.92) into (6.79), we obtain 



(6.91) 



(6.92) 



4 2 

1 4 1 











1 4 1 

2 4 



«0 
O] 



a«-i 



/(x )+^/ (1) (*o) 



/(*n-l) 



(6.93) 



=/ 

Now we have A € r(«+ 1 ) x ("+ 1 ) ) anc j we see that Aa — f of (6.93) will have a 
unique solution provided that A -1 exists. 

Matrix A is tridiagonal, and so is quite sparse (i.e., it has many zero-valued 
entries) because of the "locality" of the function S(x). This locality makes it pos- 
sible to evaluate pj,(x) in (6.74) efficiently. If we know that x e [xk, Xk+i], then 



P3(x) = a k -\S k -\{x) + a k S k (x) + a k+ \S k +\(x) + ak+iSk+lix). 



(6.94) 



We write suppg(x) = [a, b] to represent the fact that g(x) — for all x < a, and 
x > b, while g(x) might not be zero- valued for x € [a,b]. From (6.68) we may 
therefore say that suppS(x) = [—2,2], and so from (6.72), suppS^x) = [x,- — 
2h, xi + 2h], so we see that Si(x) is not necessarily zero-valued for x € [x,- — 
2h, xt + 2h]. From this, (6.94), and the fact that x,- = xq + ih, we obtain 



,k+2 



supp p 3 (x) = U^_jsupp Si(x) = [x + (* - 3)h, x Q + (k+ l)h] U 
• • • U [x + kh, x Q + (k + 4)h] 



(6.95) 



(if A,- are sets, thenU^_ A,- = A n U • • • U A m ), which "covers" the interval 
[xk, Xk+i] — [xo + kh, xq + (k + l)h\. Because the sampling grid (6.65)-(6.67), 
is uniform it is easy to establish that (via x > xo + kh) 



k = 



xo 



h 



(6.96) 



where [x] — the largest integer that is < x € R. Since suppS(x) is an interval of 
finite length, we say that S(x) is compactly supported. This locality of support is 
also a part of what makes splines useful in wavelet constructions. 
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We now consider the solution of Aa — f in either of (6.87) or (6.93). Obviously, 
Gaussian elimination is a possible method, but this is not efficient since Gaussian 
elimination is a general procedure that does not take advantage of any matrix 
structure. The sparse tridiagonal structure of A can be exploited for a more efficient 
solution. We will consider a general method of tridiagonal linear system solution 
here, but it is one that is based on modifying the general Gaussian elimination 
method. We begin by defining 
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(6.97) 



Clearly, A e r("+ 1 ) x ("+ 1 ). There is some terminology associated with tridiagonal 
matrices. The main diagonal consists of the elements a, ,, while the diagonal above 
this consists of the elements atj+i, and is often called the superdiagonal. Similarly, 
the diagonal below the main diagonal consists of the elements a,+i,;, and is often 
called the subdiagonal. 

Our approach to solving Ax — f in (6.97) will be to apply Gaussian elimination 
to the augmented linear system [A\f] in order to reduce A to upper triangular form, 
and then backward substitution will be used to solve for x. This is much the same 
procedure as considered in Section 4.5, except that the tridiagonal structure of A 
makes matters easier. To see this, consider the special case of n — 3 as an example, 
that is (A° = A,f°= f) 



[A°|/°] = 
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We may apply elementary row operations to eliminate a® in (6.98). Thus, (6.98) 
becomes 
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Now we apply elementary row operations to eliminate a 21 . Thus, (6.99) becomes 
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Finally, we eliminate o| 2 , in which case (6.100) becomes 
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We have A 3 = U, an upper triangular matrix. Thus, Ux — / 3 can be solved by 
backward substitution. The reader should write a pseudocode program to implement 
this approach for any n. Ideally, the code ought to be written such that only the 
vector /, and the main, super-, and subdiagonals of A are stored. 
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We observe that the algorithm we have just constructed will work only if a\ • ^ 
for all i = 0, 1, . . . , n. Thus, our approach may not be stable. We expect this 
potential problem because our algorithm does not employ any pivoting. However, 
our application here is to use the algorithm to solve either (6.87) or (6.93). For A in 
either of these cases, we never get a\ ■ = 0. We may justify this claim as follows. 
In Definition 6.2 it is understood that a,- ; = for i, j g Z n+ \. 

Definition 6.2: Diagonal Dominance The tridiagonal matrix A — 

\fli, ;'](', ;'=0 n € R("+ 1 ) x (' i + 1 ) is diagonally dominant if 

fl;,i > k,i-il + Kml > o 

for i =0, 1, . . . , n. 

It is clear that A in (6.87), or in (6.93) is diagonally dominant. For the algorithm 
we have developed to solve Ax — f, in general we can say 

a k 
a k+1 -a k k+hk a k (6 1021 

"k+l,k+l — a k+\,k+\ k "k,k+l> 10.1UAJ 



with a^\ k = 0, and 



a k,k+ 1 = a k,k+l (6.103) 



for k = 0, 1, . . . , n — 1. Condition (6.103) states that the algorithm does not modify 
the superdiagonal elements of A. 

Theorem 6.2: If tridiagonal matrix A is diagonally dominant, then the algo- 
rithm for solving Ax — f will not yield a\ ■ = for any i = 0, 1, . . . , n. 

Proof We give only an outline of the main idea. The complete proof needs 
mathematical induction. 
From Definition 6.2 

«o,o> l«o,-il + l«o,il = l a o,il >0, 

so from (6.102) 

1 _ o fl o,i 
a l,l — a l,l ~o~ a i,o> 
a o,o 

with aj = 0. Thus, A 1 can be obtained from A = A. Now again from Defini- 
tion 6.2 

a% > |a? | + |o? 2 | > 0, 
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so that, because < |oq j/flg | < 1, we can say that 



a\ , > a? i 



•*o,i 



•*o,o 



l«i.ol 



> l«? l + l«i 2 l 



*o,i 



^0,0 



l«i,ol 



= 1 



'0,1 



^0,0 



l«l,0l + l«l,2l 



> \a\ 2 \ = |aj >2 | = |o} >0 | + \a\ 2 \ >0, 

and A 1 is diagonally dominant. The diagonal dominance of A implies the diagonal 
dominance of A 1 . In general, we see that if A k is diagonally dominant, then A k+1 
will be diagonally dominant as well, and so A k+1 can be formed from A k for all 
k = 0, 1, . . . , n - 1. Thus, a\ ■ ^ for all i = 0, 1, . . . , w. 

If A is diagonally dominant, it will be well-conditioned, too (not obvious), and 
so our algorithm to solve Ax = / is actually quite stable in this case. 

Before considering an example of spline interpolation, we specify a result con- 
cerning the accuracy of approximation with cubic splines. This is a modified version 
of Theorem 3.13 in Burden and Faires [6], or of Theorem 4.7 in Epperson [9]. 

Theorem 6.3: If f(x) e C 4 [a, b] with m^ x(z[aM |/ (4) (x)| < M, and Pi {x) 
is the unique complete spline interpolant for f(x), then 



max I/O) - p 3 (x)\ < —Mb*. 

x€[a,b] J64 

Proof The proof is omitted, but Epperson [9] suggests referring to the article 
by Hall [10]. 

We conclude this section with an illustration of the quality of approximation of the 
cubic splines. 

Example 6.5 Figure 6.3 shows the natural and complete cubic spline inter- 
polants to the function f(x) — exp(— x 2 ) for x e [—1, 1]. We have chosen the 
nodes x^ — — 1 + jk, for k = 0, 1,2, 3, 4 (i.e., n = 4, and h — j). Clearly, 
/W(x) = — 2x exp(— x 2 ). Thus, at the nodes 

/(±1) = 0.36787944, f{±\) = 0.60653066, /(0) = 1.00000000, 



and 



f w (±l) = ^0.73575888. 
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Figure 6.3 Natural (a) and complete (b) spline interpolants (dashed lines) for the function 
f(x) = e~ x (solid lines) on the interval [— 1, 1]. The circles correspond to node locations. 
Here n = 4, with h = i, and xq = — 1. The nodes are at x k = xq + hk for k = 0, . . . , n. 

Of course, the spline series for our example has the form 

5 



Pi(x) = J^ a k S k (x), 



-l 



so we need to determine the coefficients a k from both (6.87), and (6.93). 
Considering the natural spline interpolant first, from (6.87), we have 



4 1 
1 4 1 
1 4 
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f(X2) 


«3 




- f(X3) ~ lf(X4) 



Additionally 



and 



11 11 

ao = -f(xo) = 7/( -1 )> fl 4 = -f(x 4 ) = -/(l), 
6 6 6 6 



a_l = f(x ) - 4a Q - a\, a 5 — f(x 4 ) - 4a 4 - a 3 . 
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The natural spline series coefficients are therefore 
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Now, on considering the case of the complete spline, from (6.93) we have 
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a_i — a\ — ^nf^'{xo), as — «3 + ^hf ( '(x4). 
The complete spline series coefficients are therefore 
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This example demonstrates what was suggested earlier, and that is that the com- 
plete spline interpolant tends to be more accurate than the natural spline interpolant. 
However, accuracy of the complete spline interpolant is contingent on accurate 
estimation, or knowledge of the derivatives /^'(xo) and /^'(x„). 
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PROBLEMS 



6.1. Find the Lagrange interpolant pi(t) — Y2j—(,XjLj(t) for data set 
{(-1, i), (0, 1), (1, -1)}. Find p 2J in P2 (t) = Y. )=o P2,jt j ■ 

6.2. Find the Lagrange interpolant /53(f) = ^2j = o x jLj(t) f° r data set 



{(0,1), (± 2), (1,|), (2, 



<2 ,*.„ V x, 2 „ v ^, -1)}. Find pjj in p 3 (t) = J2%o P3jt j . 
6.3. We want to interpolate x(t) at t = t\, h, and x^(t) = dx(t)/dt at t — to, £3 



using p3(t) = J2j=o P3,jt J .Let Xj = x(tj) and 
ear system of equations satisfied by (pij). 

(i)/ 



,(i) 



= x w (tj). Find the lin- 



6.4. Find a general expression for L , . (t) = dLj(t)/dt. 

6.5. In Section 6.2 it was mentioned that fast algorithms exist to solve Vander- 
monde linear systems of equations. The Vandermonde system Ap — x is 
given in expanded form as 



to 



1 t„-i 
1 t„ 
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PQ 
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Pi 
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Xn — 1 




Pn 




Xn 



(6.P.1) 



and p n (t) — Yl"j=o Pj* J interpolates the points {(tj, Xj)\k — 0, 1, 2, ... , n}. 
A fast algorithm to solve (6.P.1) is 
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for k := to n - 1 do begin 

for / := n downto to k + 1 do begin 
Xi:=(Xj-x,_i)/(ti-ti_ k _iy, 
end; 
end; 
for k :— n — 1 downto do begin 
for / := k to n - 1 do begin 

X; := X; - t k X i+ i ; 

end; 
end; 

The algorithm overwrites vector i = [io ii • • • x n ] T with the vector p — 
[po P\ ■■■ PnV ■ 

(a) Count the number of arithmetic operations needed by the fast algorithm. 
What is the asymptotic time complexity of it, and how does this compare 
with Gaussian elimination as a method to solve (6.P.1)? 

(b) Test the fast algorithm out on the system 



1 1 1 
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Po 
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1 2 4 
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1 3 9 


27 
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58 


1 4 16 


64 


_ Pi 




112 



(c) The "top" k loop in the fast algorithm produces the Newton form (Section 
6.3) of the representation for p n (t). For the system in (b), confirm that 



k-\ 



Pn(t) = ^X k Y[(t ~ti), 



k=0 1=0 

where {x^k e Z n+ i} are the outputs from the top k loop. Since in (b) 
n — 3, we must have for this particular special case 

p 3 (t) =x +xi(t- t ) + x 2 (t - t )(t - h) + x 3 (t - t )(t - t x )(t - t 2 ). 

(Comment: It has been noted by Bjorck and Pereyra that the fast algorithm 
often yields accurate results even when A is ill-conditioned.) 

6.6. Prove that for 



A n = 



to 


t 2 
l 




t\ 


t 2 


.n — \ 
' ' l \ 


tn-l 


t 2 

l n-\ 


.n — \ 
" l n-\ 


tn 


t 2 


t n — 1 



TLFeBOOK 



PROBLEMS 



287 



we have 



det(A„) = Yl ('i ~ '.0- 



0<i<j<n 



(Hint: Use mathematical induction.) 

6.7. Write a MATLAB function to interpolate the data {(tj,Xj)\j e Z„ + i} with 
polynomial p n (t) via Lagrange interpolation. The function must accept f as 
input, and return p n (t). 

6.8. Runge's phenomenon was mentioned in Section 6.2 with respect to interpo- 
lating f(t) — 1/(1 + f 2 ) on t e [—5, 5], Use polynomial p n (t) to interpolate 
/(f) at the points t k = t o + kh, k — 0, 1, . . . , n, where to — —5, and t „ — 5 
(so h — (t n - t )/n). Do this for n — 5, 8, 10. Use the MATLAB function 
from the previous problem. Plot fit) and p n (t) on the same graph for all of 
n — 5, 8, 10. Comment on the accuracy of interpolation as n increases. 

6.9. Suppose that we wish to interpolate f(t) = sinf for t e [0, jt/2] using poly- 
nomial p n (t) — Yl"j=o Pn,jt J ■ The approximation error is <?„(?) = f(t) — 
p n (t). Since \f (n) (t)\ < 1 for all t e R via (6.14), it follows that 



\e„(t)\ < 



1 



(n+l)\ 



f\\t - tt\ = Ht). 



(6.P.2) 



,=o 



Let the grid (sample) points be to = 0, t n = tt/2, and t^ — to + kh for £ = 
0, 1, . . . , n so that h — (t n - t Q )/n = ^. Using MATLAB 

(a) For « = 2, on the same graph plot /(f) — p n {t), and ±£>(f). 

(b) For n — 4, on the same graph plot /(f) — p„(t), and ±£>(f). 

Cases (a) and (b) can be separate plots (e.g., using MATLAB subplot). Does 
the bound in (6.P.2) hold in both cases? 



6.10. Consider f(t) — coshf = j[e' + e '] which is to be interpolated by P2(t) — 
S/=o P2jt' on f € [-1, 1] at the points to = -1, fo = 0, t\ = 1. Use (6.14) 
to find an upper bound on the size of the approximation error e n (t) for 
f € [—1, 1], where the size of the approximation error is given by the norm 
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6.11. For 
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it is claimed by Gautschi that 



IV 



-in , rrl±M 

I loo < max — 

0<k<n ^ \t k -t t \ 
ijtk 



(6.P.3) 



Find an upper bound on k^CV) that uses (6. P. 3). How useful is an upper 
bound on the condition number? Would a lower bound on condition number 
be more useful? Explain. 

6.12. For each function listed below, use divided difference tables (recall Example 
6.3) to construct the degree n Newton interpolating polynomial for the spec- 
ified points. 

(a) f(t) = 4t , fo = 0, t\ — 1, H — 3. Use n = 2. 

(b) f(t) — coshf, to — — 1, t\ — -j-, ii — 2> 

(c) f(t) — Int, to = 1, t\ = 2, t2 = 3. Use n = 2. 



_1 i 2 = i f 3 = 1. Use n = 3. 



(d) /(0 = 1/(1 4 
6.13. Prove Eq. (6.33). 



f), to = 0,t\ = A, ?2 = 1, ?3 = 2. Use « = 3. 



6.14. Consider Theorem 6.1. For n = 2, find /^(x) and ^^(x) as direct-form (i.e., 
a form such as (6.4)) polynomials in x. Of course, do this for all k — 0, 1, 2. 
Find Hermite interpolation polynomial p2n+\(x) for n — 2 that interpolates 
/(x) = .y/x at the points xo = 1, x\ — 3/2, X2 = 2. 

6.15. Find the Hermite interpolating polynomial p$(x) to interpolate f(x) — e x at 
the points xo = 0, x\ — 0.1, X2 = 0.2. Use MATLAB to compare the accu- 
racy of approximation of ps(x) to that of p^ix) given in Example 6.3 (or 
Example 6.4). 

6.16. The following matrix is important in solving spline interpolation problems 
[recall (6.87)]: 
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• 1 
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Suppose that D n — det(A„). 

(a) Find D\, D2, Dj, and D4 by direct (hand) calculation using the basic 
rules for computing determinants. 

(b) Show that 



D 



n+2 



4D„ 



D„=0 
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(i.e., the determinants we seek may be generated by a second-order dif- 
ference equation). 

(c) For suitable constants a, ft e R, it can be shown that 

D n =a(2 + v / 3)"+ y 8(2- V3f (6.P.4) 

in e N). Find a and [J. 

[Hint: Set up two linear equations in the two unknowns a and fJ using 

(6. P. 4) for n = 1, 2 and the results from (a).] 

(d) Prove that D„ > for all n e N. 

(e) Is A„ > for all n e N? Justify your answer. 

[//mf: Recall part (a) in Problem 4.13 (of Chapter 4).] 

6.17. Repeat Example 6.5, except use 

fM = rh> 

and x € [—5, 5] with nodes Xk — — 5 + \k for k — 0, 1, 2, 3, 4 (i.e., « = 4, 
and /; = |). How do the results compare to the results in Problem 6.8 (assum- 
ing that you have done Problem 6.8)? Of course, you should use MATLAB 
to "do the dirty work." You may use built-in MATLAB linear system solvers. 

6.18. Repeat Example 6.5, except use 



f(x) = Vl+x. 

Use MATLAB to aid in the task. 

6.19. Write a MATLAB function to solve tridiagonal linear systems of equations 
based on the theory for doing so given in Section 6.5. Test your algorithm 
out on the linear systems given in Example 6.5. 
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7.1 INTRODUCTION 

In this chapter we consider the problem of finding x to satisfy the equation 

/0O = (7.1) 

for arbitrary /, but such that f(x) e R. Usually we will restrict our discussion 
to the case where x e R, but x e C can also be of significant interest in practice 
(especially if / is a polynomial). However, any x satisfying (7.1) is called a root 
of the equation. Such an x is also called a zero of the function /. More generally, 
we are also interested in solutions to systems of equations: 

f (xo,xi,...,x„-i) = 0, 
fi(xo,xi, ...,x n -\) = 0, 

fn-l(xo,Xl, ...,X n -l) — 0. (7.2) 

Again, fi(xo, x\, . . . , x n -\) e R, and we will assume Xk e R for all ;', k e Z„. Var- 
ious solution methods will be considered such the bisection method, fixed-point 
method, and the Newton-Raphson method. All of these methods are iterative in 
that they generate a sequence of points (or more generally vectors) that converge 
(we hope) to the desired solution. Consequently, ideas from Chapter 3 are rele- 
vant here. The number of iterations needed to achieve a solution with a given 
accuracy is considered. Iterative procedures can break down (i.e., fail) in various 
ways. The sequence generated by a procedure may diverge, oscillate, or display 
chaotic behavior (which in the past was sometimes described as "wandering behav- 
ior"). Examples of breakdown phenomena will therefore also be considered. Some 
attention will be given to chaotic phenomena, as these are of growing engineering 
interest. Applications of chaos are now being considered for such areas as cryp- 
tography and spread-spectrum digital communications. These proposed applications 
are still controversial, but at the very least some knowledge of chaos gives a deeper 
insight into the behavior of nonlinear dynamic systems. 
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The equations considered in this chapter are nonlinear, and so are in contrast 
with those of Chapter 4. Chapter 4 considered linear systems of equations only. The 
reader is probably aware of the fact that a linear system either has a unique solution, 
no solution, or an infinite number of solutions. Which case applies depends on the 
size and rank of the matrix in the linear system. Chapter 4 emphasized the handling 
of square and invertible matrices for which the solution exists, and is unique. In 
Chapter 4 we saw that well-defined procedures exist (e.g., Gaussian elimination) 
that give the solution in a finite number of steps. 

The solution of nonlinear equations is significantly more complicated than the 
solution of linear systems. Existence and uniqueness problems typically have no 
easy answers. For example, e 2x + 1 = has no real- valued solutions, but if we allow 
x e C, then x — jknj for which k is an odd integer (since e 2x + 1 = e-' nk + 1 = 
cos(£jt) + j sin(A:7T) + 1 = cos(kjt) = —1 + 1 = as k is odd). On the other hand 

e~' - sin(f) = 

also has an infinite number of solutions, but for which t e R (see Fig. 7.1). How- 
ever, the solutions are not specifiable with a nice formula. In Fig. 7.1 the solutions 
correspond to the point where the two curves intersect each other. 

Polynomial equations are of special interest. 1 For example, x 2 + 1 = has only 
complex solutions x — ±j . Multiple roots are also possible. For example 

x 3 - 3x 2 + 3x - 1 = (x - l) 3 = 

has a real root of multiplicity 3 at x = 1. We remark that the methods of this 
chapter are general and so (in principle) can be applied to find the solution of 
any nonlinear equation, polynomial or otherwise. But the special importance of 
polynomial equations has caused the development (over the centuries) of algorithms 
dedicated to polynomial equation solution. Thus, special algorithms exist to solve 

n 

Pn(.x) = J^p n , k X k = 0, (7.3) 

k=0 

Why are polynomial equations of special interest? There are numerous answers to this. But the reader 
already knows one reason why from basic electric circuits. For example, an unforced RLC (resistance x 
inductance x capacitance) circuit has a response due to energy initially stored in the energy storage 
elements (the inductors and capacitors). If x(t) is the voltage drop across an element or the current 
through an element, then 

d n x(t) d n ~ l x(t) dx(t) 

The coefficients a^ depend on the circuit elements. The solution to the differential equation depends on 
the roots of the characteristic equation: 

a„X" +a n _iX"~ l H \-a\X + aQ = 0. 
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Figure 7.1 Plot of the individual terms in the equation /(f) = e r — sin(f) = 0. The infi- 
nite number of solutions possible corresponds to the point where the plotted curves intersect. 



and these algorithms do not apply to general nonlinear equations. Sometimes 
these are based on the general methods we consider in this chapter. Other times 
completely different methods are employed (e.g., replacing the problem of find- 
ing polynomial zeros by the equivalent problem of finding matrix eigenvalues as 
suggested in Jenkins and Traub [22]). However, we do not consider these special 
polynomial equation solvers here. We only mention that some interesting references 
on this matter are Wilkinson [1] and Cohen [2], These describe the concept of ill- 
conditioned polynomials, and how to apply deflation procedures to produce more 
accurate estimates of roots. The difficulties posed by multiple roots are considered 
in Hull and Mathon [3]. Modern math-oriented software tools (e.g., MATLAB) 
often take advantage of theories such as described in Refs. 1-3, (as well as in 
other sources). In MATLAB polynomial zeros may be found using the roots and 
mroots functions. Function mroots is a modern root finder that reliably determines 
multiple roots. 



7.2 BISECTION METHOD 

The bisection method is a simple intuitive approach to solving 

fix) = 0. (7.4) 

It is assumed that fix) e R, and that x e R. This method is based on the following 
theorem. 

Theorem 7.1: Intermediate Value Theorem If f\[a, b] — > R is continuous 
on the closed, bounded interval [a, b], and yo e R is such that f(a) < yo < fib), 
then there is an xo e [a,b] such that fixo) — yo- In other words, a continuous 
function on a closed and bounded interval takes on all values between /(a) and 
f{b) at least once. 
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The bisection method works as follows. Suppose that we have an initial interval 
[ao, bo] such that 

f(a )f(b ) < 0, (7.5) 

which means that /(ao) and f(bo) have opposite signs (i.e., one is positive while 
the other is negative). By Theorem 7.1 there must be a p e (ao, bo) so that f(p) = 
0. We say that [ao, bo] brackets the wot p. Suppose that 

Po = 3(00 + ^0)- (7.6) 

This is the midpoint of the interval [ao, bo]- Consider the following cases: 

1. If f(po) — 0, then p — po and we have found a root. We may stop at this 
point. 

2. If / '(ao) / (po) < 0, then it must be the case that p e [ao, po], so we define 
the new interval [a\, b\] — [ao, po]- This new interval brackets the root. 

3. If f (po) f (bo) < then it must be the case that p e [po, bo] so we define 
the new interval [a\, b\] — [po, bo]- This new interval brackets the root. 

The process is repeated by considering the midpoint of the new interval, which is 

pi = \(ai+bi), (7.7) 

and considering the three cases again. In principle, the process terminates when 
case 1 is encountered. In practice, case 1 is unlikely in part because of the effects 
of rounding errors, and so we need a more practical criterion to stop the process. 
This will be considered a little later on. For now, pseudocode describing the basic 
algorithm may be stated as follows: 

input [ao, bo] which brackets the root p; 
PO := (a + b )/2; 
k:=0; 

while stopping criterion is not met do begin 
if fia^fipk) < then begin 
a k+-\ -= a k- 
b k+1 -=Pk- 
end; 
else begin 
%+1 -=Pk- 
b k+1 -= b k- 
end; 
p k+ 1 :=(a k+ i +b k+ i)/2; 
k :=k+-\; 
end; 

When the algorithm terminates, the last value of Pk+\ computed is an estimate 
of p. 
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We see that the bisection algorithm constructs sequence (p n ) — (po, pi, P2, ■ ■ ■) 

such that 

lim p n = p, (7.8) 

n— >oo 

where p„ is the midpoint of [a n , b n ], and f(p) = 0. Formal proof that this process 
works (i.e., yields a unique p such that f(p) — 0) is due to the following theorem. 

Theorem 7.2: Cantor's Intersection Theorem Suppose that ([a^, b^]) is a 
sequence of closed and bounded intervals such that 

[ao, bo] D[ai,bi]D ■■■ D [a„, b„]D ■■■ , 
with lim n ^oo(/) n — a n ) — 0. There is a unique point p e [a„,b n ] for all n e Z + : 



P)[a n >&n] = {/?}• 



n=0 



The bisection method produces (p n ) such that a n < p n < b n , and p € [a n , b n ] for 

l 2< 



all n e Z + . Consequently, since p n — \{a n + b n ) 



\Pn-p\<\b„-a„\<—^- (7.9) 

for n e Z + , so that limj^oo p n — p. Recall that / is assumed continuous on [a, b], 
so lim„^oo f(pn) — f(p)- So now observe that 

1 1 

\Pn ~a„\< —\b - a\, \b„ - p n \ < —\b - a\ (7.10) 

so, via \x — y\ — \(x — z) — (y — z)\ < \x — z\ + \y — z\ (triangle inequality), we 
have 

1 1 1 

\p-a„\ <\p- p n \ + \p n -a n \ < —(b-a)+ —(b-a) = — j(b-a), 

(7.11a) 
and similarly 



\p-b n \<\p- p n \+ \p n -b n \<—^(b-a). (7.11b) 



1 

2" 



Thus 

lim a n — lim b n — p. (7.12) 

At each step the root p is bracketed. This implies that there is a subsequence [of (p n )] 
denoted (x n ) converging to p so that f(x n ) > for all n e Z + . Similarly, there is a 
subsequence (y n ) converging to p such that f(y n ) < for all n e Z + . Thus 

f(p) = lim f(x„) > 0, f(p) = lim f(y n ) < 0, 



TLFeBOOK 



BISECTION METHOD 



295 



which implies that f(p) — 0. We must conclude that the bisection method produces 
sequence (/?„) converging to p such that f(p) — 0. Hence, the bisection method 
always works. 

Example 7.1 We want to find 5 1 / 3 (cube root of five). This is equivalent to 
solving the equation f(x) — x 3 — 5 = 0. We note that /(l) = —4 and /(2) = 3, 
so we may use [ao, bo] = [a,b] = [1,2] to initially bracket the root. We remark 
that the "exact" value is 5 1 / 3 = 1.709976 (seven significant figures). Consider the 
following iterations of the bisection method: 



[oo,6 ] = [l,2], 
[a\,b\] = [po, bo], 
[a 2 ,b 2 ] = [oi, p\], 
[03,63] = [P2,b 2 ], 
[04,64] = [P3,h], 
[a 5 ,6 5 ] = [04, pa\ 



po = 1.500000, 
pi = 1.750000, 
p 2 = 1.625000, 
p 3 = 1.687500, 
p 4 = 1.718750, 
/?5 = 1.703125, 



f(p ) = -1.625000 
f(pi) = 0.359375 
/Q, 2 ) = -0.708984 
f(pj) = -0.194580 
/0? 4 ) = 0.077362 
/(p 5 ) = -0.059856 



We see that |p 5 - p\ = |1.703125 - 1 .7099761 = 0.006851. From (7.9) 

6-o 2-1 1 

| P5 -/>I< — = — = ? = 0.0312500. 

The exact error certainly agrees with this bound. 

When should we stop iterating? In other words, what stopping criterion should 
be chosen? Some possibilities are (for e > 0) 



(7.13a) 

(7.13b) 
(7.13c) 

(7.13d) 



We would stop iterating when the inequalities are satisfied. Usually (7.13d) is 
recommended. Condition (7.13a) is not so good as termination depends on the size 
of the nth interval, while it is the accuracy of the estimate of p that is of most 
interest. Condition (7.13b) requires knowing p in advance, which is not reasonable 
since it is p that we are trying to determine. Condition (7.13c) is based on f(p n ), 
and again we are more interested in how well p n approximates p. Thus, we are left 
with (7.13d). This condition leads to termination when p n is relatively not much 
different from p n -\. 



2\Pn ^n) 


< e, 




\Pn ~ P\ <€, 




f(Pn) < e, 




Pn - Pn-\ 
Pn 


< € 


(Pn # 0) 
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How may we characterize the computational efficiency of an iterative algorithm? 
In Chapter 4 the algorithms terminated in a finite number of steps (with the excep- 
tion of the iterative procedures suggested in Section 4.7 which do not terminate 
unless a stopping condition is imposed), so flop counting was a reasonable mea- 
sure. Where iterative procedures such as the bisection method are concerned, we 
prefer 

Definition 7.1: Rate of Convergence Suppose that (x n ) converges to 0: 
liirin^oo x„ — 0. Suppose that (p n ) converges to p, i.e., linin^oo p„ — p. If there 
is a K € R, but K > 0, and N e Z+ such that 

\Pn ~ P\ < K\x n \ 

for all n > N , then we say that (/?„) converges to p with rate of convergence 0(x n ). 

This is an alternative use of the "big O" notation that was first seen in Chapter 4. 
Recalling (7.9), we obtain 

\Pn-p\<(b-a)^, (7.14) 

so that K — b — a, x n = ^r, and N — 0. Thus, the bisection method generates 
sequence (/?„) that converges to p (with f(p) — 0) at the rate 0(1/2"). 
From (7.14), if we want \p n — p\ < e, then we may choose n so that 



\Pn-p\<-^r<e, 



implying 2" > (b — a)/e, or we may choose n so 

' b — a 



log 2 



(7.15) 



where \x~\ — smallest integer greater than or equal to x. This can be used as an 
alternative means to terminate the bisection algorithm. But the conservative nature 
of (7.15) suggests that the algorithm that employs it may compute more iterations 
than are really necessary for the desired accuracy. 



7.3 FIXED-POINT METHOD 

Here we consider the Banach fixed-point theorem [4] as the theoretical basis for a 
nonlinear equation solver. Suppose that X is a set, and T\X — > X is a mapping of 
X into itself. A fixed point of T is an x e X such that 



Tx=x. (7.16) 
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For example, suppose X = [0, 1] C R, and 

Tx=jx(l-x). (7.17) 

We certainly have Tx e [0, 1] for any x e X. The solution to 

X — 2*0 — X) 

(i.e., to Tx — x) is x — 0. So T has fixed point x — 0. (We reject the solution 
x = -1 since -1 £ [0, 1].) 

Definition 7.2: Contraction Let X — (X, d) be a metric space. Mapping 
T\X — > X is called a contraction (or a contraction mapping, or a contractive 
mapping) on X if there is an a € R such that < a < 1 , and for all x , y e X 

d(Tx,Ty) <ad(x,y). (7.18) 

Applying T to "points" x and y brings them closer together if T is a contractive 
mapping. If a — 1, the mapping is sometimes called nonexpansive. 

Theorem 7.3: Banach Fixed-Point Theorem Consider the metric space X — 
(X, d), where X g 0. Suppose that X is complete, and T\X — >■ X is a contraction 
on X. Then r has a unique fixed point. 

Proof We must construct (x„), and show that it is Cauchy so that (x„) con- 
verges in X (recall Section 3.2). Then we prove that x is the only fixed point of 
T. Suppose that xo € X is any point from X. Consider the sequence produced by 
the repeated application of T to xq: 

xo, x\ — Txq, X2 — Tx\ — T xo, ■ . ■ , x n = Tx n -\ — T n xo, .... (7.19) 

From (7.19) and (7.18), we obtain 

d(x k+ i,x k ) = d(Txk, Txk-i) 

< ad(xk,Xk-i) 

— ad(Txk-\, Tx k _ 2 ) 

< a 2 d(x k -i,x k - 2 ) 



<a k d(xi,x ). (7.20) 
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Using the triangle inequality with n > k, from (7.20) 

d(x k ,x n ) < d(x k , x k +i) + d(x k+ i, x k+2 ) -\ \-d{x n -i,x n ) 

< (a k + a k+1 ■■■+ a n - 1 )d(x Q , xi) 



k 



\-a 



n—k 



i-« 



-d(xo, x\) 



(where the last equality follows by application of the formula for geometric series, 
seen very frequently in previous chapters). Since < a < 1, we have 1 — a n ~ k < 1. 

Thus 

a k 

d(x k ,x n ) < d(xo,x\) (7.21) 

1 — a 

(n > k). Since 0<o<l, 0<1— a < 1, too. In addition, d(xo, x\) is fixed (since 
we have chosen xq). We may make the right-hand side of (7.21) arbitrarily small 
by making k sufficiently big (keeping n > k). Consequently, (x n ) is Cauchy. X is 
assumed to be complete, so x n -> x e X. 
From the triangle inequality and (7.18) 

d(Tx, x) < d(x, x k ) + d(x k , Tx) 

< d(x, x k ) + ad(x k -\, x) (recall that Tx — x) 

< € (any e > 0) 

if A: — > oo, since x n — > x. Consequently, d(x, Tx) — implies that Tx — x (recall 
(Ml) in Definition 1.1). Immediately, x is a fixed point of T. 

Point x is unique. Let us assume that there is another fixed point x, i.e., Tx — x. 
But from (7.18) 

d(x, x) — d(Tx, Tx) < ad(x, x), 

implying d(x, x) — because < a < 1. Thus x — x [(Ml) From Definition 1.1 
again]. 

Theorem 7.3 is also called the contraction theorem, a special instance of which 
was seen in Section 4.7. The theorem applies to complete metric spaces, and so is 
applicable to Banach and Hilbert spaces. (Recall that inner products induce norms, 
and norms induce metrics. Hilbert and Banach spaces are complete. So they must 
be complete metric spaces as well.) 

Note that we define T°x — x for any x e X. Thus T° is the identity mapping 
(identity operator). 

Corollary 7.1 Under the conditions of Theorem 7.3 sequence (x n ) from x n — 
T"xq [i.e., the sequence in (7.19)] for any xq e X converges to a unique x e X 
such that Tx — x. We have the following error estimates (i.e., bounds): 
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1. Prior estimate: 



2. Posterior estimate: 



a k 



d(xk,x) < d(xo,xi) (7.22a) 

\ — a 



a 

d(xk,x) < d(xk,Xk-i). (7.22b) 

1 — a 

Proof The prior estimate is an immediate consequence of Theorem 7.3 since 
it lies within the proof of this theorem [let n — > oo in (7.21)]. 

Now consider (7.22b). In (7.22a) let k = 1, yo = x o< an d yi = x i- Thus, 
d(xi, x) — d(yi, x), d(xo, x\) — d(yo, y\), and so 

a 

d(yi,x)<- d(y ,yi). (7.23) 

1 — a 

Let yo — Xk-i, so y\ — Tyo — Txk-i — x\ and (7.23) becomes 

a 

d(x k ,x) < d(xk,Xk-i), 

1 — a 

which is (7.22b). 

The error bounds in Corollary 7.1 are useful in application of the contraction the- 
orem to computational problems. For example, (7.22a) can estimate the number of 
iteration steps needed to achieve a given accuracy in the estimate of the solution to 
a nonlinear equation. The result would be analogous to Eq. (7.15) in the previous 
section. 

A difficulty with the theory so far is that T\X -> X is not always contractive 
over the entire metric space X, but only on a subset, say, Y C X. However, a basic 
result from functional analysis is that any closed subset of X is complete. Thus, for 
any closed Y C X, there is a fixed point x e Y C X, and x„ — > x ix n — T h xq for 
suitable xq e Y). For this idea to work, we must choose xo € Y so that x n e Y for 
all n € Z+. 

What does "closed subset" (formally) mean? A neighborhood of x e X (metric 
space X) is the set N € (x) — {y\d(x, y) < e, e > 0} C X. Parameter e is the radius 
of the neighborhood. We say that x is a limit point of Y C X if every neighborhood 
of x contains y ^ x such that ye Y. Y is closed if every limit point of Y belongs 
to Y. These definitions are taken from Rudin [5, p. 32]. 

Example 7.2 Suppose that X — R, Y = (0, 1) C X. We recall that X is a 
complete metric space if d(x, y) = \x — y\ (x, y e X). Is Y closed? Limit points 
of Y are x — and x = 1. [Any x e (0, 1) is also a limit point of Y.] But e" Y, 
and 1 f" Y. Therefore, Y is not closed. On the other hand, Y — [0, 1] is a closed 
subset of X since the limit points are now in Y. 
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Fixed-point theorems (and related ideas) such as we have considered so far have 
a large application range. They can be used to prove the existence and uniqueness 
of solutions to both integral and differential equations [4, pp. 314-326]. They can 
also provide (sometimes) computational algorithms for their solution. Fixed-point 
results can have applications in signal reconstruction and image processing [6], 
digital filter design [7], the interpolation of bandlimited sequences [8], and the 
solution to so-called convex feasibility problems in general [9]. However, we will 
consider the application of fixed-point theorems only to the problem of solving 
nonlinear equations. 

If we wish to find p e R so that 

fip) = 0, (7.24) 

then we may define g(x) (g(x)\R. — > R) with fixed point p such that 

g(x) = x-f(x), (7.25) 

and we then see g(p) = p — f(p) = p =$■ fip) — 0. Conversely, if there is a 
function g(x) such that 

g(p) = P, (7.26) 

then 

/(*) = x - g(x) (7.27) 

will have a zero at x — p. So, if we wish to solve f(x) — 0, one approach would 
be to find a suitable g(x) as in (7.27) (or (7.25)), and find fixed points for it. 

Theorem 7.3 informs us about the existence and uniqueness of fixed points for 
mappings on complete metric spaces (and ultimately on closed subsets of such 
spaces). Furthermore, the theorem leads us to a well-defined computational algo- 
rithm to find fixed points. At the outset, space X — R with metric d(x, y) — \x — y\ 
is complete, and for us g and / are mappings on X. So if g and / are related 
according to (7.27), then any fixed point of g will be a zero of /, and can be 
found by iterating as spelled out in Theorem 7.3. However, the discussion follow- 
ing Corollary 7.1 warned us that g may not be contractive onX = R but only on 
some subset Y C X. In fact, g is only rarely contractive on all of R. We therefore 
usually need to find Y — [a, b] C X — R such that g is contractive on Y, and then 
we compute x n — g n xo e Y for n e Z + . Then x n — >• x, and f(x) — 0. We again 
emphasize that x n e Y is necessary for all n. If g is contractive on [a, b] C X, 
then, from Definition 7.2, this means that for any x, y e [a,b] 

\g(x)-g(y)\<a\x-y\ (7.28) 

for some real- valued a such that < a < 1. 

Example 7.3 Suppose that we want roots of fix) — kx 2 + (1 — k)x — 
(assume X > 0). Of course, this is a quadratic equation and so may be easily 
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solved by the usual formula for the roots of such an equation (recall Section 4.2). 
However, this example is an excellent illustration of the behavior of fixed-point 
schemes. It is quite easy to verify that 

f(x)=x-Xx(l-x). (7.29) 

We observe that g(x) is a quadratic in x, and also 

g m (x) = (2x-l)k, 

so that g(x) has a maximum at x — j for which g(j) — A/4. Therefore, if we 
allow only < X < 4, then g(x) e [0, 1] for all x e [0, 1]. Certainly [0, 1] C R. A 
sketch of g(x) and of y — x appears in Fig. 7.2 for various X. The intersection of 
these two curves locates the fixed points of g on [0, 1], 

We will suppose Y — [0, 1]. Although g(x) e [0, 1] for all x € [0, 1] under the 
stated conditions, g is not necessarily always contractive on the closed interval 
[0, 1]. Suppose X — j; then, for all x e [0, 1], g(x) e [0, 4] (see the dotted line 
in Fig. 7.2, and we can calculate this from knowledge of g). The mapping is 
contractive for this case on all of [0, 1]. [This is justified below with respect to 
Eq. (7.30).] Also, g(x) = x only for x — 0. If we select any xq e [0, 1], then, for 
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Figure 7.2 Plot of g(x) = Xx{\ — x) for various X, and a plot of y = x. The places where 
y = x and g(x) intersect define the fixed points of g(x). 
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x n — g n xo, we can expect *„ — > 0. For example, suppose *o — 0.7500; then the 
first few iterates are 

xq = 0.7500 

*i = 0.5xo(l - *o) = 0.0938 
x 2 = 0.5*i(l-*i) = 0.0425 
* 3 = 0.5*2(1 - *2) = 0.0203 
* 4 = 0.5*3(1 -*3) = 0.0100 
* 5 = 0.5*4(1 - * 4 ) = 0.0049 



The process is converging to the unique fixed point at * = 0. 

Suppose now that X — 2; then g(x) — 2*(1 — *), and now g(x) has two fixed 
points on [0, 1], which are * = 0, and * = 5. For all * € [0, 1] we have g(x) e 
[0, j], but g is not contractive on [0, 1], For example, suppose * = 0.8, y = 0.9, 
theng(.8) = 0.3200, and g(.9) = 0.1800. Thus, |* - y\ = 0.1, but |g(*) - g(y)\ = 
0.14 > 0.1. On the other hand, suppose that *o = 0.7500; then the first few iter- 
ates are 

*o = 0.7500 

*i = 2* (1 - * ) = 0.3750 
* 2 = 2*i(l -*i) = 0.4688 
* 3 = 2* 2 (1 - xi) = 0.4980 
* 4 = 2*3(1 -xi) = 0.5000 
x 5 = 2* 4 (1 - * 4 ) = 0.5000 



This process converges to one of the fixed points of g even though g is not 
contractive on [0, 1]. 
From (7.28) 



a > 



g(x) - g(y) 



(* # y), 



x-y 
from which, if we substitute g(x) — Xx(l — *), and g(y) — Xy(l — y), then 

a = supA.|l — * — y\. (7.30) 

If A. = |. then a — j. If X — 2, we cannot have a, so that < a < 1. If 
g(x) — 2*(1 — *) (i.e., X — 2 again), but now instead Y — [0.4, 0.6], then, for all 
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x € Y = [0.4, 0.6], we have g(x) e [0.48, 0.50] C Y, implying g\Y -+ Y. From 
(7.30) we have for this situation a — 0.4. The mapping g is contractive on Y. 
Suppose x = 0.450000; then 

x = 0.450000 

xi = 2*o(l - *o) = 0.495000 
x 2 = 2xi(l - xi) = 0.499950 
x 3 = 2x 2 (l - x 2 ) = 0.500000 
x 4 = 2x 3 (l - x 3 ) = 0.500000 



The reader ought to compare the results of this example to the error bounds from 
Corollary 7.1 as an exercise. 

More generally, and again from (7.28), we have (for x, y e Y — [a, b] C R) 

g(x) - g(y) 



a = sup 



x-y 



(7.31) 



Now recall the mean-value theorem (i.e., Theorem 3.3). If g(x) is continuous on 
[a, b], and g^Hx) is continuous on (a, b), then there is a £ € (a, b) such that 

g (i) (?) _ g(P)-g(fl) 



b — a 



Consequently, instead of (7.31) we may use 

a — sup \g^'(x) 

x£(a,b) 



(7.32) 



Example 7.4 We may use (7.32) to rework some of the results in Example 7.3. 
Since g (1) (x) = A(2x - 1), if Y = [0, 1], and A. = ±, then 



a = sup 

*€(0,1) 



x 

2 



If k = 2, then a = 2. If now Y = [0.4, 0.6] with k = 2, then 

a = sup |4x - 2| = 0.4. 

x€(0.4,0.6) 

Now suppose that k — 2.8, and consider Y — [0.61, 0.67], which contains a fixed- 
point of g (see Fig. 7.2, which contains a curve for this case). We have 

a = sup |5.6x - 2.8| = 0.9520, 
xe(0.6i,0.67) 
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and if x € Y, then g(x) e [0.619080, 0.666120] C Y so that g\Y ->• Y, and so g is 
contractive on Y. Thus, we consider the iterates 

x = 0.650000 

xi = 2.8x (l - xo) = 0.637000 
x 2 = 2.8xi (1 - xi ) = 0.647447 
x 3 = 2.8x 2 (l - x 2 ) = 0.639126 
x 4 = 2.8x 3 (1 - x 3 ) = 0.645803 
x 5 = 2.8x 4 (l - x 4 ) = 0.640476 



The true fixed point (to 6 significant figures) is x = 0.642857 (i.e., x n — > x). We 
may check these numbers against the bounds of Corollary 7.1. Therefore, we con- 
sider the distances 

d(x 5 ,x) = 0.002381, d(x , xi) = 0.013000, d(x 5 , x 4 ) = 0.005327, 

and from (7.22a) 

a k 

d(x 5 ,x) < d(x ,xi) = 0.211780, 

1 — a 



and from (7.22b) 

<i(x 5 ,x) < — — d(x 5 ,x 4 ) = 0.105652. 
1 — a 

These error bounds are very loose, but they are nevertheless consistent with the 
true error d(x 5 , x) = 0.002381. 

We have worked with g(x) — Xx(l — x) in the previous two examples 
(Example 7.3 and Example 7.4). But this is not the only possible choice. It may 
be better to make other choices. 

Example 7.5 Again, assume f(x) — Xx 2 + (1 — X)x — as in the previous 
two examples. Observe that 

Xx — (X — l)x 



implies that 



A- A /^— ^x 1/2 = g(x). (7.33) 
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If k = 4, then f(x) = for x = and for * = f . For g(x) in (7.29), g (1) (*) = 
8x — 4, and g^(|) = 2. We cannot find a closed interval Y containing x — | on 
which g is contractive with g\Y —*■ Y (the slope of the curve g(x) is too steep in 
the vicinity of the fixed point). But if we choose (7.33) instead, then 



2 * v~> 4 ,1/2' 

and if F = [0.7, 0.8], then for x e Y, we have g(x) € [0.7246, 0.7746] C Y, and 

V3 1 
a = sup — — 0.5175. 

jte(0.7,0.8) 4 ^/X 

So, g in (7.33) is contractive on Y. We observe the iterates 

x = 0.7800 

V3 



*1 = 


= — X 


= 0.7649 


*2 = 


v3 1/2 
2 l 


= 0.7574 


*3 = 


v3 1/2 
~-— X 2 


= 0.7537 



VJ 1/2 

X 

2 



x 4 = — x 3 7 = 0.7518 



a/3 
x 5 = ^-*j /2 = 0.7509, 



which converge to x — I (i.e., x„ — >• x). 



7.4 NEWTON-RAPHSON METHOD 



This method is yet another iterative approach to finding roots of nonlinear equations. 
In fact, it is a version of the fixed-point method that was considered in the previous 
section. However, it is of sufficient importance to warrant separate consideration 
within its own section. 

7.4.1 The Method 

One way to derive the Newton-Raphson method is by a geometric approach. The 
method attempts to solve 

f(x) = (7.34) 
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Liney-f(p„) = f< 1 >(p n )(x-p n ) 
[tangent to point (p„, f(p n )] 




Figure 7.3 Geometric interpretation of the Newton-Raphson method. 



(x € R, and f(x) e R) by approximating the root p (i.e., f(p) = 0) by a succession 
of x intercepts of tangent lines to the curve y = f(x) at x the current approximation 
to p. Specifically, if p n is the current estimate of p, then the tangent line to the 
point (p n , f(p n )) on the curve y = f(x) is 



y = f(Pn) + f m (Pn)(x-Pn) 



(7.35) 



(see Fig. 7.3). The next approximation to p is x 
from (7.35) 



Pn+i such that y — 0, that is, 



= f(p n ) + f m (Pn)(Pn 



+ 1 



Pn), 



so for n e Z + 



/?«+! = Pn 



f(Pn) 
f m (PnY 



(7.36) 



To start this process off, we need an initial guess at p, that is, we need to select pq. 
Clearly this approach requires f^Hpn) # for all n; otherwise the process will 
terminate. Continuation will not be possible, except perhaps by choosing a new 
starting point pq. This is one of the ways in which the method can break down. 
It might be called premature termination. (Breakdown phenomena are discussed 
further later in the text.) 

Another derivation of (7.36) is as follows. Recall the theory of Taylor series 
from Section 3.5. Suppose that f(x), /^'(x), and f^ 2 \x) are all continuous on 
[a, b]. Suppose that p is a root of f(x) — 0, and that p n approximates p. From 
Taylor's theorem [i.e., (3.71)] 



f(x) = f(Pn) + f W (Pn)(x 



Pn) + \f {2) mx 



Pnf 



(7.37) 
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where | = f (x) e (p n , x). Since fip) = from (7.37), we have 

= fip) = /(/>„) + f m (PnKp ~ Pn) + jf (2) (i;Kp - Pnf- (7.38) 

If I p — Pn I is small, we may neglect that last term of (7.38), and hence 

0^f(Pn) + f W (Pn)(p-Pn), 

implying 

P^ Pn- } ■ (7.39) 

f iV) (Pn) 

We treat the right-hand side of (7.39) as p n +\, the next approximation to p. Thus, 
we again arrive at (7.36). The assumption that (p — p n ) 2 is negligible is important. 
If p„ is not close enough to p, then the method may not converge. In particular, 
the choice of starting point po is important. 

As already mentioned, the Newton-Raphson method is a special instance of the 
fixed-point iteration method, where 

fix) 
g (x) = x- J n , . (7.40) 

So 

Pn + l = giPn) (7.41) 

for n e Z + . Stopping criteria are (for e > 0) 

\Pn ~ Pn-\\ < e, (7.42a) 

\fiPn)\ < e, (7.42b) 

Pn ~ Pn-\ 



Pn * 0. (7.42c) 

Pn 

As with the bisection method, we prefer (7.42c). 

Theorem 7.4: Convergence Theorem for the Newton-Raphson Method 

Let / be continuous on [a, b]. Let f^ l \x), and f^ 2 \x) exist and be continuous for 
all x e (a, b). If p e [a, b] with f(p) — 0, and f^(p) ^ 0, then there is a S > 
such that (7.36) generates sequence (/?„) with p„ — > p for any po e [p — S, p + S], 

Proof We have 

Pn+l = giPn), n eZ + 

with g(;c) = x - 4$- . We need Y = [p - S, p + 8] C R with g|T -^ Y, and g 
is contractive on Y . (Then we may immediately apply the convergence results from 
the previous section.) 
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Since f^'(p) # 0, and since f^(x) is continuous at p, there will be a S\ > 
such that f^\x) ^ for all x e [p — Si, p + S\] C [a, b], so g is defined and 
continuous for x G [p — Si, p + Si], Also 

(1) [f W (x)] 2 -f(x)f (2 Hx) f(x)f<- 2 \x) 

8 W [/ (1) (X)] 2 U {1) (X)] 2 

for x € [p — Si, p + Si]. In addition, f^(x) is continuous on (a, £>), so g^(x) is 
continuous on [p — Si, p + Si]. We assume f(p) = 0, so 

8 (p> if m ( P )i 2 

We have g^\x) continuous at x — p, implying that lim^^p g^\x) — g^ l \p) — 0, 
so there is a 5 > with < S < S{ such that 

\g m (x)\<a< 1 (7.43) 

for x e [p — S, p + S] — Y for some a such that 0<« < l.If x e 7, then by the 
mean- value theorem, there is a £ € (x, p) such that 

\g(x) -p\ = \g(x) - g(p)\ = |g (1) (£)||* - p\ 
< a\x — p\ < \x — p\ < S, 

so that we have g(x) e Y for all x e Y. That is, g\Y — > 7. Because of (7.43) 

a = sup|g (1) (x)|, 

and < a < 1, so that g is contractive on Y. Immediately, sequence (p n ) from 
Pn+i = gOn) for all po € F converges to p (i.e., /?„ -> /?). 

Essentially, the Newton-Raphson method is guaranteed to converge to a root if 
po is close enough to it and / is sufficiently smooth. Theorem 7.4 is weak because 
it does not specify how to select po. 

The Newton-Raphson method, if it converges, tends to do so quite quickly. 
However, the method needs f^Hpn) as well as f(p n ). It might be the case that 
f^Hpn) requires much effort to evaluate. The secant method is a variation on the 
Newton-Raphson method that replaces f^(Pn) with an approximation. Specifi- 
cally, since 

,(1), . ,. f(x)-f(Pn) 
f { '(Pn)= lim , 

x^p„ X- p n 
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f m (Pn) « /(p«-i) /W _ (7 44) 

Pn-1 - Pn 



This is the slope of the chord that connects the points (p n -\, f{p n -\)), and 
(Pn, f(Pn)) on the graph of fix). We may substitute (7.44) into (7.36) obtaining 
(for n e N) 

(Pn~ Pn-l)f(Pn) ,„... 

P«+i = /?« - — — — -. (7.45) 

f(Pn) ~ f(Pn-l) 

We need po and p\ to initialize the iteration process in (7.45). Usually these are 
chosen to bracket the root, but this does not guarantee convergence. If the method 
does converge, it tends to do so more slowly than the Newton-Raphson method, but 
this is the penalty to be paid for avoiding the computation of derivatives f^\x). 
We remark that the method of false position is a modification of the secant method 
that is guaranteed to converge because successive approximations to p (i.e., p n -\ 
and p n ) are chosen to always bracket the root. But it is possible that convergence 
may be slow. We do not cover this method in this book. We merely mention that 
it appears elsewhere in the literature [10-12]. 



7.4.2 Rate of Convergence Analysis 

What can be said about the speed of convergence of fixed-point schemes in gen- 
eral, and of the Newton-Raphson method in particular? Consider the following 
definition. 

Definition 7.3: Suppose that (p n ) is such that p n -> p with p n ^ p (n e Z + ). 
If there are X > 0, and S > such that 

.. \Pn+l ~ P\ , 
hm j- — X, 

n ^°° \p n - pr 

then we say that (p n ) converges to p of order S with asymptotic error constant X. 
Additionally 

• If S = 1, we say that (/?„) is linearly convergent. 

• If 5 > 1, we have superlinear convergence. 

• If S — 2, we have quadratic convergence. 

From this definition, if n is big enough, then 

\ Pn+l -p\^X\p n -p\ s . (7.46) 

Thus, we would like S to be large and X to be small for fast convergence. Since 
S is in the exponent, this parameter is more important than X for determining the 
rate of convergence. 
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Consider the fixed-point iteration 

Pn+\ = g(Pn), 

where g satisfies the requirements of a contraction mapping (so the Banach fixed- 
point theorem applies). We can therefore say that 

.. \Pn + \~P\ .. \g(Pn)-g(p)\ ._.„ 

hm = hm , (7.47) 

n^OO \p n - p\ n^oo \p n - p\ 

so from the mean-value theorem we have % n between p n and p for which 

g(Pn)-g(p)=g m (t;n)(.Pn-p), 

which, if used in (7.47), implies that 

Hm lP " +1 ~ g! = Iim|,«(fe,)|. (7.48) 

n^oo \p n — p\ n^oo 

Because f„ is between p n and p and p n — >• /?, we must have £„ -> p as well. Also, 
we will assume g^Hx) is continuous at p, so (7.48) now becomes 

lim 'f +1 "f =| g ^(p)|, (7.49) 

which is a constant. Applying Definition 7.3, we conclude that the fixed-point 
method is typically only linearly convergent because 8=1, and the asymptotic 
error constant is A = \g ( - 1 \p)\, provided g^ l \p) ^ 0. However, if g^ip) = 0, we 
expect faster convergence. It turns out that this is often the case for the Newton- 
Raphson method, as will now be demonstrated. 

The iterative scheme for the Newton-Raphson method is (7.36), which is a 
particular case of fixed-point iteration where now 

fix) 

g (x) =x- J n \ , (7.50) 

f (l Hx) 



and for which 



(i)/..x f(x)f (2) (x) 



(x) = J —^- 1— (7.51) 

[/<1)(jc)]2 



so that g^(p) = because f(p) = 0. Thus, superlinear convergence is anticipated 
for this particular fixed-point scheme. Suppose that we have the Taylor expansion 



g(x) = g(p) + g w (p)(x -p) + £gW(f )(* - p) 



I (2), 
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for which f is between p and x. Since g(p) — p and g^Hp) — 0, this becomes 

g{x) = p+\ g W{i;)(x-p) 2 . 

For x — p n , this in turn becomes 

Pn+l = g0>») = p + ^ (2) fe)(p - Pnf (7.52) 

for which % n lies between p and p n . Equation (7.52) can be rearranged as 
p n+l - p= \g (2) {$n){p ~ Pn) 2 , 

and so 

lim 'f"* 1 "' 1 - V>(rtl. (7-53) 

n^oo \p n - p\± 2 

since f„ — »• /?, and we are also assuming that g^Hx) is continuous at p. Imme- 
diately, 8 = 2, and the Newton-Raphson method is quadratically convergent. The 
asymptotic error constant is plainly equal to j\g ( - 2 \p)\, provided g^ 2 Hp) ^ 0. If 
g( 2 \p) = 0, then an even higher order of convergence may be expected. It is 
emphasized that these convergence results depend on g(x) being smooth enough. 
Given this, since the convergence is at least quadratic, the number of accurate dec- 
imal digits in the approximation to a root approximately doubles at every iteration. 

7.4.3 Breakdown Phenomena 

The Newton-Raphson method can fail to converge (i.e., break down) in vari- 
ous ways. Premature termination was mentioned earlier. An example appears in 
Fig. 7.4. This figure also illustrates divergence, which means that [if f(p) = 0] 

lim \p„ - p\ = oo. 

In other words, the sequence of iterates (p n ) generated by the Newton-Raphson 
method moves progressively farther and farther away from the desired root p. 

Another failure mechanism called oscillation may be demonstrated as follows. 
Suppose that 

f{x) = x 3 - 2x + 2, f m (x) = 3x 2 - 2. (7.54) 

Newton's method for this specific case is 

Pi -lPn + 2 2pl - 2 
Pn + r = Pn- ^ _ - = j-r^. (7.55) 

If we were to select po — as a starting point, then 

Po = 0, p\ = 1, P2 - 0, p 3 = 1, £>4 = 0, P5 = l,.... 
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Figure 7.4 Illustration of breakdown phenomena in the Newton-Raphson method; here, 
f(x) = xe~ x . The figure shows premature termination if p n = x = 1 since /^ '(1) = 0. 
Also shown is divergence for the case where p$ > 1. (See how the x intercepts of the 
tangent lines for x > 1 become bigger as x increases.) 



More succinctly, p n — ^(1 — (—1)"). The sequence of iterates is oscillating; it 
does not diverge, and it does not converge, either. The sequence is periodic (it may 
be said to be period 2), and in this case quite simple, but far more complicated 
oscillations are possible with longer periods. 

Premature termination, divergence, and oscillation are not the only possible 
breakdown mechanisms. Another possibility is that the sequence (j>„) can be 
chaotic. Loosely speaking, chaos is a nonperiodic oscillation with a complicated 
structure. Chaotic oscillations look a lot like random noise. This will be considered 
in more detail in Section 7.6. 



7.5 SYSTEMS OF NONLINEAR EQUATIONS 

We now extend the fixed-point and Newton-Raphson methods to solving nonlinear 
systems of equations. We emphasize two equations in two unknowns (i.e., two- 
dimensional problems). But much of what is said here applies to higher dimensions. 

7.5.1 Fixed-Point Method 

As remarked, it is easier to begin by first considering the two-dimensional problem. 
More specifically, we wish to solve 



/o(*o,*i)=0, /i(xn,xi) = 0, 
which we will assume may be rewritten in the form 

xq - fo(xo,xi) = 0, xi - fi(xo, xi) = 0, 



(7.56) 



(7.57) 



and we see that solving these is equivalent to finding where the curves in (7.57) 
intersect in R 2 (i.e., [xq x\\ t e R 2 ). A general picture is in Fig. 7.5. As in the 
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Figure 7.5 Typical curves in R corresponding to the system of equations in (7.57). 



case of one-dimensional problems, there will usually be more than one way to 
rewrite (7.56) in the form of (7.57). 



Example 7.6 Suppose 



fo(x ,x\) = x -x 
f\(xo,xi) =x\ -xl 



1*2 = 



/oOo, x\) — xl + \x\, 



x i =0=> fi(xo, xi) = x 



We see that 



U2 



ixt - x = o =► o*o - 4r + jxf = j 



1\2 i U2 



(an ellipse in R ), and 



xi =0 



(xi + i) 2 = -1 



(a hyperbola in R 2 ). These are plotted in Fig. 7.6. The solutions to the system 
are the points where the ellipse and hyperbola intersect each other. Clearly, the 
solution is not unique. 

It is frequently the case that nonlinear systems of equations will have more than 
one solution just as this was very possible and very common in the case of a single 
equation. 

Vector notation leads to compact descriptions. Specifically, we define x — 
[x xi] T € R 2 , f (x) = fo(x ,xi), fi(x) = fi(x ,xi), and 



F(x) = 



foixo, X\) 
f\{xo,xi) 



fo(x) 
fl(x) 



(7.58) 
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Figure 7.6 The curves of Example 7.6 (an ellipse and a hyperbola). The solutions to the 
equations in Example 7.6 are the points where these curves intersect. 



Then the nonlinear system in (7.57) becomes 

x — F (x) , 

and the fixed point ~p e R 2 of F satisfies 

P = F(p). 

We recall that R 2 is a normed space if 



(7.59) 



(7.60) 



mi 2 



= lZ X ! 2 = X > 



(7.61) 



(x e R ). We may consider a sequence of vectors (x n ) (i.e., x n e R for n e Z + ), 
and it converges to x iff 

lim | \x„ —x\\ — 0. 
n— >oo 

We recall from Chapter 3 that R 2 with the norm in (7.61) is a complete space, so 
every Cauchy sequence (x n ) in it will converge. 

As with the scalar case considered in Section 7.3, we consider the sequence of 
iterates (~p n ) such that 

Tn + \ = F (Pn)> n e Z+ - 

The previous statements (and the following theorem) apply if R 2 is replaced by R m 
(m > 2); that is, the space can be of higher dimension to accommodate m equations 
in m unknowns. Naturally, for R m the norm in (7.61) must change according to 
| |x| | 2 = Y^k=o x k- The following theorem is really a special instance of the Banach 
fixed-point theorem seen earlier. 
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Theorem 7.5: Suppose that 1Z is a closed subset of R 2 , F\7Z — > TZ, and F is 
contractive on TZ; then 

x — F (x) 

has a unique solution /J € R. The sequence (p n ), where 

-p n+l = F(p n ), -p en, neZ+ (7.62) 



is such that 



and 



lim \\p n -p\\=0 



\\p n -p\\<-, IIPi-^oll. ( 7 - 63 ) 

1 — a 

where ||F (pj) — F(75 2 )|| 5 a ll7?i — 7*2 1 1 f° r an y ~Pi'~P2 e ^> anc ^ < « < 1. 

Proof As noted, this theorem is really a special instance of the Banach fixed- 
point theorem (Theorem 7.3), so we only outline the proof. 

It was mentioned in Section 7.3 that any closed subset of a complete metric 
space is also complete. R 2 with norm (7.61) is a complete metric space, and since 
TZ C R 2 is closed, 1Z must be complete. According to Theorem 7.3, F has a unique 
fixed point ~p e TZ [i.e., F(p) — ~p], and sequence (~p n ) from ~p~ n+ \ — F(p n ) (with 
7?o € ^> ar, d n € Z + ) converges to 75. The error bound 

a" 

\\Pn~ P\\ < -, \\Pi-Pq\\ 

1 — a 
is an immediate consequence of Corollary 7.1. 

A typical choice for 7\L would be the bounded and closed rectangular region 

TZ — {[xqx\\ \oq < xq < bo, a\ < x\ < b\). (7.64) 

The next theorem applies the Schwarz inequality of Chapter 1 to the estimation 
of a. It must be admitted that applying the following theorem is often quite difficult 
in practice, and in the end it is often better to simply implement the iterative 
method in (7.62) and experiment with it rather than go through the laborious task 
of computing a according to the theorem's dictates. However, exceptions cannot 
be ruled out, and so knowledge of the theorem might be helpful. 



Theorem 7.6: Suppose that TZ C R 2 is as defined in (7.64); then, if 

'(to)\(!*)\(*L)\(yL) 

\dx J \dxiJ \dxoJ \dxij 



a — max 



2"|l/2 

(7.65) 
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we have 

||F(*i)-F(*2)||<«||3fi-3f2|| 
for all x~i, x 2 € H. 

Proof We use a two-dimensional version of the Taylor expansion theorem, 
which we will not attempt to justify in this book. 

Given xi — [x^o x\\] T , x 2 — [x 2 fi x 2 ,i] , there is a point f e 1Z on the line 
segment that joins x\ to X2 such that 

F(xi) = F(J 2 ) + ,F (1) (f)(xi-X2) 



F(x 2 ) 



3/bte) 3/b(f) 

dxQ 3xi 

3/l (£) 3/i (f) 



|F(xi)-F(x- 2 )H 2 = 



- 3*o 
3/o(§) 



dx 



1 -" 



*1,0 - *2,0 
XI, 1 -X2,l 



3x 

' 3/i(S) 

. 3*o 

3/o(g) 

3x 



Oi,0 - x 2 ,o) + 



3/o(S) 

3xi 



(Xl,l -X 2 ,l) 



(Xl,0 - *2,o) + 



3/o«) 



3/i (f) 
3xi 



(Xll -X 2 ,l) 



3xi 



'W + ( 



3/i (?) 



3xo / \ 3xi 
via the Schwarz inequality (Theorem 1.1). Consequently 



11*1 ~x 2 \ 



|*i -x 2 | 



|F(xi)-F(x 2 )|| 2 < 



/ 3/ (g) 

V 3*0 



3/i(£) 

3x 



3/o(g) 

3xi 

3/i (g) 

3xi 



|xi -x 2 || 2 



< < max 



3/oOD 

3xo 



3/i(*) x2 



3x 

= a 2 ||x 1 -x 2 || 



3/oCO 

3x 1 



3/i(*) x2 
3x 1 



|xi -x 2 | 
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As with the one-dimensional Taylor theorem from Chapter 3, our application of it 
here assumes that F is sufficiently smooth. 

To apply Theorem 7.5, we try to select F and 1Z so that < a < 1. As already 
noted, this is often done experimentally. The iterative process (7.62) can give us 
an algorithm only if we have a stopping criterion. A good choice is (for suitable 
e > 0) to stop iterating when 

\\P„ -Pn-l\\ - / n <n ct* 
— — < e, Pn ^ 0. (7.66) 

\\Pn\\ 

It is possible to use norms in (7.66) other than the one defined by (7.61). (A 
Chebyshev norm might be a good alternative.) Since our methodology is often to 
make guesses about F and 1Z, it is very possible that a given guess will be wrong. 
In other words, convergence may never occur. Thus, a loop that implements the 
algorithm should also be programmed to terminate when the number of iterations 
exceeds some reasonable threshold. In this event, the program must also be written 
to print a message saying that convergence has not occurred because the upper limit 
on the allowed number of iterations was exceeded. This is an important example 
of exception handling in numerical computing. 

Example 7.7 If we consider /o(xo> xi) and f\(xQ, x\) as in Example 7.6, then 
the sequence of iterates for the starting vector 7J = [1 l] r is 

Pi fi = 1.2500, p u = 0.0000 

P2,o = 1.5625, p2,\ — 1.5625 

P3 = 3.0518, p 3> i = 0.0000 

p A fi = 9.3132, P4,i = 9.3132 



This vector sequence is not converging to the root near [0.9 0.5] r (see Fig. 7.6). 
Many experiments with different starting values do not lead to a solution. So we 
need to change F, 



We may rewrite xq - 


- X|-j — 4-^1 — " 9-S 








2 1 2 — J0\ x Q> x l) 

X Q + 4*1 


i we may rewrite x\ 


— Xq + x\ — as 




x 2 
X\ = , - = /i(x , *l)- 



1+X! 
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These redefine F [recall (7.58) for the general form of F], For this choice of F 
and again with 75 = [1 l] r , the sequence of vectors is 

pi,o = 0.8000, pi,i = 0.5000 

p 2 ,o = 0.9110, p 2 ,i = 0.4627 

P3j0 = 0.9480, p 3 ,i = 0.5818 

p 4 ,o = 0.9140, p 4 ,i = 0.5682 



pi4,o = 0.9189, pi4,i = 0.5461 

In this case the vector sequence now converges to the root with four-decimal-place 
accuracy by the 14th iteration. 



7.5.2 Newton-Raphson Method 

Consider yet again two equations in two unknowns 

/o(*o.*i) = °> 

/i(*0,*i) = 0, (7.67) 

each of which defines a curve in the plane (again x — [xo x\\ T e R 2 ). Solutions 
to (7.67) are points of intersection of the two curves in R 2 . We will denote a point 
of intersection by ~p — [po P\\ T e R 2 , which is a root of the system (7.67). 

Suppose thatxo = [xo,o xo,i] r is an initial approximation to the root p. Assume 
that fo and f\ are smooth enough to possess a two-dimensional Taylor series expan- 
sion, 

3/o 
f (x , X\) = /o(*0,0. x 0,l) + - — 0*0,0, *o,i)(*o - *o,o) 

3xo 

, 9 /o, w . , ! f 3 2 /o(xo,o,x j) ,2 

+ 7 — (*0,07*0,l)(*l -X0,l)+ — { —^ (xo -X ,o) 

3xi 2! [ dx£ 

9 2 /o(xo,o,x ,i) 

+ 2 — (x - x ,o)(xi - x ,i) 

dxoaxi 

3 2 /o(xo,o,x ,i) / 2 f 



dxi 



-(xi -x ,i) > + • 



3/i 
/i(xo,xi) = /i(xo,o, xo,i) + - — (xo,o,xo,i)(x -xo,o) 

3xo 

, 3/i 1 [3 2 /i(xo,o,x j) 2 

+ 7 — (*0,0>*0,i)(*i -xo,i)+ — { — 5 (xo-xo^o) 

3xi 2! ^ 9xq 
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,3 2 /l(*0,0>*0,l) , .. . 

-(x -xo,o)(xi -*o,i) 



3xo3xi 

(x\ -x ,i) \ + 



9 2 /i(x ,o,*o,i) / > 



9x 2 j 

which have the somewhat more compact form using vector notation as 

_ _ 9/o (To) 9/o (To) 

/o(T) = /o(*o) H ; (*o - *o,o) ^ ^ ( Xl ~ x o,i) 

9xo 9xi 

4 1 \ d 2 f (x ) , , 2 , „ 3 2 /oC*o) , ., . 

+ — \ 5 — (*o - *o,o) + 2— — ^ — C*o - *o,o)(*i - *o,i) 

2! ^ 9xq dxooxi 



3 2 /o(*o) 

9x 2 



(*i-x ,i) 2 [ + --- (7.68a) 



_ _ 9/i (To) 9/i (xo) 

/lW = /i(*o) H ; (xo - x ,o) H - (*i - *o,i) 

9xo 9xi 

1 [ 9 2 /i(xq) / , 2 , o 3 2 /i(T ) / 

+ ^ 1 — :T~? — w> ~ x °.°) + 2 ^ — 5 — (^0 ~ *o,o)(*i - *o,i) 
2! [ 9x^ 9x 9x! 

9 2 /i(x ) / 2 



9x 2 



(*i-x ,i) z +•••• (7.68b) 



If xq is close to p, then from (7.68), we have 



_ _ 9/o(^o) 9/o(xo) 

= fo(p) « /o(*o) H r Oo - *o,o) + — t 0>i - *o,i) 

9xo 9xi 

2 «,/-=■„■> a2 £,/■=■„ 



1 3Vo0to), . ,2 , 9 2 /o(To) 



(Po-^0,0) +2— — - — (po-^o,o)(Pi -*o,i) 



2! 9xi ' 9xq9x 



v 



, 8 M X 0h .2 I , , nt -Q X 

H —5 — (pi-^0,1) M (7.69a) 

9x 2 J 

_ _ 3/i (To) 9/i (To) 

0= f\(p) « fi(xo)-\ (po-^o,o)H (p\ -*o,i) 

9xo 9xi 

, 1 f 9 2 /i(To) . . 2 , „ 3 2 /i(*o) , 

G>o-*o,o) + 2— — - — (po - xo,o)(pi - xo,\) 



2! 9x 2 ' 9x 9x 



v 



9 /l(*0), .2 I , ,"7*010, 

— —5 — (Pi-^0,1) H (7.69b) 

9x 2 j 
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If we neglect the higher-order terms (second derivatives and higher-order deriva- 
tives), then we obtain 

9/oC*o) 9/o(*o) _ 

— (po - x ,o) H (pi - x 0i i) «s _/ ( Xo ), (7.70a) 

dxo ax\ 

9/iC*o) 9/iC*o) _ 

— (pQ-x 0fi )-\ (pi -x ,i) « -/i(x ). (7.70b) 

dxo ox\ 

As a shorthand notation, define 

dfi (In) 
3x 7 - 

so (7.70) becomes 

(po - *o,o)/b,o(*o) + (Pi - *o,i)./b,i(*o) « -/o(*o)> (7.72a) 

(f>0 - *o,o)/i,o(*o) + (Pi - *o,i)/i,i(*o) % -/i (*(>)• (7.72b) 

Multiply (7.72a) by /i,i(x"o), and multiply (7.72b) by /o.iCco). Subtracting the 
second equation from the first results in 

(po - xo,o)[/o,o(*o)/i,i(*o) - /i,o(*o)/o,i(*o)] 

^ -/o(*o)/i,i(*o) + /i(*o)/o,i (*())■ (7.73a) 

Now multiply (7.72a) by /i.oCxoX an d multiply (7.72b) by /o.oCxo)- Subtracting 
the second equation from the first results in 

(p\ -*o,i)[/o,i(*o)/i,o(*o) - /o,o(*o)/u(*o)] 

^ -/o(*o)/i,o(*o) + /i(*o)/o,o(*o)- (7.73b) 

From (7.73b), we obtain 

, ~ fo(xo) f\,\(xo) + f\(xo) fo,\(xo) ,hha^ 

Po ^ xoo~\ = — — =—, (7.74a) 

/o,o(*o) /uw - /o,i(*o)/i,o(*o) 

, -/i(*o)/o,o(*o) + /o(*o)/i,o(*o) „„,,, 

Pi % *o i H = — — =—■ (7.74b) 

foM x o)fi,i(xo) - /o,i(*o)/i,o(*o) 

We may assume that the right-hand side of (7.74) is the next approximation to ~p 

-/o/i,i + /i/o,i 



■*i,0 % *o,o + 

*1,1 «*0,1 + 



/o,o/i,i - /o,i/i,o 
-/i/o,o + /o/i,o 



/o,o/i,i — /o,i/i,o 



(7.75a) 
(7.75b) 



TLFeBOOK 



SYSTEMS OF NONLINEAR EQUATIONS 321 



(~x\ — [xi, 0*1,1] ), where the functions and derivatives are to be evaluated at xo. 
We may continue this process to generate (x„) for n e Z + (so in general x n — 
[x n ,o *«,i] r ) according to 



-/o(*»)/i,i(*») + /i(*n)/o,i(*») 
fo,o( J n)fi,i(x n ) - fo,l(x„)flfi(x n )' 

~fl(Xn)f0,0(Xn) + /o(*«)/l,o(*») 
fo,o(x n )fl,l(x n ) - fo,l(Xn)fl,o(Xn) 

As in the previous subsection, we define 



Xn+1,0 — X„fi + 



Xn+1,1 — X n ,\ 



(7.76a) 
(7.76b) 



Also 



F(x n ) = 



F m (x n ) = 



fo(x n ,Q, *n,l) 
fl(Xn,0, X n ,\) 



f0,0(Xn) fo,l(Xn) 
/l,o(*n) f\,\(x n ) 



fo(x„) 

fl(x~n) 



= Jf(x„), 



(7.77) 



(7.78) 



which is the Jacobian matrix Jf evaluated at x = x n - We see that 

1 



[j F (x n )r l = 



f0,0(Xn)fl,l(x n ) ~ f0,l(Xn)fl,0(Xn) 



fl,l(x n ) -fo,l(Xn) 
f\fi(x n ) /b,o(*n) 



(7.79) 

so in vector notation (7.76) becomes 

x n +\ =x„- [J F (x„)] _1 F(x„) (7.80) 

for n e Z + . If ~x n e R m (i.e., if we consider m equations in m unknowns), then 



Jf(x„) = 



fo,o( x n) fo,l(.X„) 

flfiiXn) f\,\(x„) 



fo,m-l(x n ) 
J\,m—{ \Xfi) 



Jm—l,0yXn) Jm—\,\\Xn) Jm—l,m—l\Xn) 



(7.81a) 



and 



F(x n ) 



MXn) 
fl(x n ) 



Jm — l \Xn) 

Of course, x n = [x nfi x n ,\ ••• x„, m -i] r e R m . 



(7.81b) 
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Equation (7.80) reduces to (7.46) when we have only one equation in one 
unknown. We see that the method will fail if Jf(x n ) is singular at x n . As in 
the one-dimensional problem of Section 7.4, the success of the method depends 
on picking a good starting point x~o. If convergence occurs, then it is quadratic 
as in the one-dimensional (i.e., scalar) case. It is sometimes possible to force the 
method to converge even if the starting point is poorly selected, but this will not 
be considered here. The computational complexity of the method is quite high. If 
x n e R m , then from (7.80) and (7.81), we require m 2 + m function evaluations, 
and we need to invert an m x m Jacobian matrix at every iteration. We know from 
Chapter 4 that matrix inversion needs 0(m 3 ) operations. Ill conditioning of the 
Jacobian is very much a potential problem as well. 

Example 7.8 Refer to the examples in Section 7.5.1. In Example 7.6 there we 
wanted to solve 

Mxq,xi) = x -Xq - \ x \ =0, /i(*o. x\) =x\ -x 2 + x 2 — 0. 
Consequently 

JQ\ x n) = X n fi x n Q 4 x n p /l (X n ) = X n i X n q + X n j, 

and the derivatives are 

fo,o(x n ) — 1 - 2x„,o, fi,o(xn) — -2x„fi 
1 

"2 

Via (7.76), the desired equations are 



fo,\(x n ) = -\x n ,l, h,\{x n ) = 1 +2x n ,\. 



X n +\fi — X n fi 

-{x nfi - x 2 Q - \xl A )(\ + 2x„,i) + (x„,i - x 2 Q + xl x ){-\x n ,\) 
(1 - 2x nfi )(\ + 2x„j) - x n!Q x nt i 

~(x n ,\ -xl fi + xl x )(\ -2x„, ) + {x„fi-xl Q - ^x^ 1 )(-2x„ i0 ) 
(1 - 2x„ i0 )(l + 2x„ ; i) - x„, ^n,i 

If we execute the iterative procedure in (7.82), we obtain 

x ,o = 0.8000, x ,i = 0.5000 

xi, = 0.9391, x u = 0.5562 

x 2 ,o = 0.9193, x 2 ,i = 0.5463 

x 3j0 = 0.9189, x 3>1 = 0.5461 



(7.82a) 



(7.82b) 
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We see that the answer is correct to four decimal places in only three iterations. 
This is much faster than the fixed-point method seen in Example 7.7. 

7.6 CHAOTIC PHENOMENA AND A CRYPTOGRAPHY APPLICATION 

Iterative processes such as x n +\ = g(x n ) (which includes the Newton-Raphson 
method) can converge to a fixed point [i.e., to x such that g(x) — x], or they 
can fail to do so in various ways. This was considered in Section 7.4.3. We are 
interested here in the case where (x n ) is a chaotic sequence, in which case g is 
often said to be a chaotic map. Formal definitions exist for chaotic maps [13, 
p. 50]. However, these are rather technical. They are also difficult to apply except 
in relatively simple cases. We shall therefore treat chaos in an intuitive/empirical 
(i.e., experimental) manner for simplicity. 

In Section 7.3 we considered examples based on the logistic map 

g{x) = Xx{l-x) (7.83) 

(recall Examples 7.3-7.5). Suppose that X = 4. Figure 7.7 shows two output 
sequences from this map for two slightly different initial conditions. Plot (a) shows 
for xq — Xq— 0.745; while plot (b), xq — Xq = 0.755. We see that |*g - x$\ — 
0.01, yet after only a few iterations, the two sequences are very different from each 
other. This is one of the distinguishing features of chaos: sensitive dependence 
of the resulting sequence to minor changes in the initial conditions. For X — 4, 
we have g\[0, 1] -> [0, 1], so divergence is impossible. Chaotic sequences do not 
diverge. They remain bounded; that is, there is a M e R such that < M < oo 
with \x n \ < M for all n e Z + . But the sequence does not converge, and it is not 
periodic, either. In fact, the plots in Fig. 7.7 show that the elements of sequence 
(x n ) seem to wander around rather aimlessly (i.e., apparently randomly). This wan- 
dering behavior has been observed in the past [14, p. 167], but was not generally 
recognized as being a chaotic phenomenon until more recently. 

It has been known for a very long time that effective cryptographic systems 
should exploit randomness [15]. Since chaotic sequences have apparently random 
qualities, it is not surprising that they have been proposed as random-number gen- 
erators (or as pseudo-random-number generators) for applications in cryptography 
[16, 17]. However, it is presently a matter of legitimate controversy regarding just 
how secure a cryptosystem based on chaos can be. One difficulty is as follows. 
Nominally, a chaotic map g takes on values from the set of real numbers. But if 
such a map is implemented on a digital computer, then, since all computers are 
finite-state machines, any chaotic sequence will not be truly chaotic as it will even- 
tually repeat. Short period sequences are cryptographically weak (i.e., not secure). 
There is presently no effective procedure (beyond exhaustive searching) for deter- 
mining when this difficulty will arise in a chaos-based system. This is not the only 
problem (see p. 1507 of Ref. 17 for others). 

Two specific chaos-based cryptosy stems are presented by De Angeli et al. [18] 
and Papadimitriou et al. [19]. (There have been many others proposed in recent 
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Figure 7.7 Output sequence from the logistic map g(x) = 4x(l— x) with initial conditions 
xq = 0.745 (a) and xq = 0.755 (b). Observe that although these initial conditions are close 
together, the resulting sequences become very different from each other after only a few 
iterations. 



years.) Key size is the number of different possible encryption keys available in the 
system. Naturally, this number should be large enough to prevent a codebreaker 
(eavesdropper, cryptanalyst) from guessing the correct key. However, it is also well 
known that a large key size is most definitely not a guarantee that the cryptosystem 
will be secure. Papadimitriou et al. [19] demonstrate that their system has a large 
key size, but their security analysis [19] did not go beyond this. Their system [19] 
seems difficult to analyze, however. In what follows we shall present some analysis 
of the system in De Angeli et al. [18], and see that if implemented on a digital 
computer, it is not really very secure. We begin with a description of their system 
[18]. Their method in [18] is based on the Henon map (see Fig. 7.8), which is a 
mapping defined on R 2 according to 



XQ,n + l = 1 -«*0,n + x l,n 
*l,n+l = /**0,n 



(7.84) 



for n e Z + (so [xo.h^i.h] 7 e R 2 )> an ^ which is known to be chaotic in some neigh- 
borhood of 

a = 1.4, /3 = 0.3 (7.85) 
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Figure 7.8 Typical state sequences from the Henon map for a = 1.45 and fi = 0.25, with 
initial conditions xqq = —0.5 and x\ o = 0.2. 

so that constants a and /3 form the encryption key for the system. No choice in the 
allowed neighborhood is a valid key. An immediate problem is that there seems 
to be no detailed description of which points are allowed. A particular choice of 
key should therefore be tested to see if the resulting sequence is chaotic. In what 
follows the output sequence from the map is defined to be 



y n = XQ,n ■ 



(7.86) 



The vector x n — [xq h x\ n ] T is often called a state vector. The elements are state 
variables. 

The encryption algorithm works by mixing the chaotic sequence of (7.86) with 
the message sequence, which we shall denote by (s n ). The mixing (described below) 
yields the cyphertext sequence, denoted (c„). A problem is that the receiver must 
somehow recover the original message (s n ) from the cyphertext (c„) using knowl- 
edge of the encryption algorithm, and the key {a, fi}. 

Consider the following mapping: 



%«+l = 1 -ay n +xi,„, 

xi,n+i = Py n - 



(7.87) 



Note that this mapping is of the same form as (7.84) except that xo,„ is replaced 
by y„, but from (7.86) these are the same (nominally). The mapping in (7.84) 
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represents a physical system (or piece of software) at the transmitter, while (7.87) 
is part of the receiver (hardware or software). Now define the error sequences 



v x i,n — x i,n 



(i € {0, 1}) 



for which it is possible to show [using (7.84), (7.86), and (7.87)] that 



8x0,n + l 



Sxo, n 

8xi „ 



(7.88) 



(7.89) 



= A 



= 


"01" 




"01" 




Sxi,„ 


= 


" " 




We observe that A 2 — 0. Matrix A is an example of a nilpotent matrix for this 
reason. This immediately implies that the error sequences go to zero in at most 
two iterations (i.e., two steps): 



<5*l,n+2 



This fact tells us that if (y n ) is generated at the transmitter, and sent over the 
communications channel to the receiver, then the receiver may perfectly recover 2 
the state sequence of the transmitter in not more than two steps. This is called dead- 
beat synchronization. The system in (7.87) is a specific example of a nonlinear 
observer for a nonlinear dynamic system. There is a well-developed theory of 
observers for linear dynamic systems [20]. The notion of an observer is a control 
systems concept, so we infer that control theory is central to the problem of applying 
chaotic systems to cryptographic applications. 

All of this suggests the following algorithm for encrypting a message. Assume 
that we wish to encrypt a length N message sequence (s n ); that is, we only have 
the elements sq,Si, . . . , sn-2, $N-i- 



1. The transmitter generates and transmits yo, y\ according to (7.84) and (7.86). 
The transmitter also generates yi, V3, . . . , y^, yN+i, but these are not trans- 
mitted. 

2. The transmitter sends the cyphertext sequence 



yn+2 



(7.90) 



for n = 0, 1 , . . . , N — 1 . Of course, this assumes we never have s„ — 0. 

Equation (7.90) is not the only possible means of mixing the message and the 
chaotic sequence together. The decryption algorithm at the receiver is as follows: 

This assumes a perfect communications channel (which is not realistic), and that rounding errors in 
the computation are irrelevant (which will be true if the receiver implements arithmetic in the same 
manner as the transmitter). 
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1. The receiver regenerates the transmitter's state sequence using (7.87), its 
knowledge of the key, and the sequence elements yo,y\. Specifically, for 
n — 0, 1 compute 

*0,n+l = 1 -uyl + h,n, 

*i,«+i = Py n (7.91a) 

while for n — 2,3, . . . , N — I, N compute 

X0,n + \ = 1 ~0Cxl n +X\, n , 

X\,n + \ = /*%«■ (7.91b) 

Recover y n for n — 2,3, . . . , N, N + I according to 

?«=%«■ (7.92) 

2. Recover the original message via 

V„_1_T 

(7.93) 







yn+2 
s n — 

Cn 


where n = 0, 1 , . . 


., N - 


-2, N - 1. 



The initial states at the transmitter and receiver are arbitrary; that is, any xo,o, *i,0 
and x\fi may be selected in (7.84) and (7.91a). The elements yo, y\, and the cypher- 
text are sent over the channel. If these are lost because of a corrupt channel, then the 
receiver should request retransmission of this information. It is extremely impor- 
tant that the transmitter resend the same synchronizing elements yo, yi and not a 
different pair. The reason will become clear below. 

We remark that since we are assuming that the algorithms are implemented 
on a digital computer, so each sequence element is a binary word of some form. 
The synchronizing elements and cyphertext are thus a bitstream. Methods exist to 
encode such data for transmission over imperfect channels such that the probability 
of successful transmission can be made quite high. These are called error control 
coding schemes. 

In general, the receiver will fail to recover the message if (1) the way in which 
arithmetic is performed by the receiver is not the same as at the transmitter, (2) the 
channel corrupts the transmission, or (3) the receiver does not know the key {a, ft}. 
Item 1 is important since the failure to properly duplicate arithmetic operations at 
the receiver will cause machine rounding errors to accumulate and prevent data 
recovery. This is really a case of improper synchronization. The plots of Figs. 7.9- 
7.11 illustrate some of these effects. 

It is noteworthy that even though the receiver may not be a perfect match to the 
transmitter, some of the samples (at the beginning of the message) are recovered. 
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E 
< 



(a) 




50 60 

Time step 



100 




(b) 



40 50 60 

Time step 



100 



Figure 7.9 Transmitted (a) and reconstructed (b) message sequences. The sinusoidal mes- 
sage sequence (s n ) is perfectly reconstructed at the receiver; because the channel is perfect, 
the receiver knows the key (here a = 1.4, /J ^ 0.3), and arithmetic is performed in identical 
fashion at both the transmitter and receiver. 



This is actually an indication that the system is not really very secure. We now 
consider security of the method in greater detail. 

What is the key size? This question seems hard to answer accurately, but a 
simple analysis is as follows. Suppose that the Henon map is chaotic in the 
rectangular region of the afi plane defined by a € [1.4 — Aa, 1.4+ Aa], and 
P e [0.3 - A0, 0.3 + Aft], with Aa, Ap > 0. We do not specifically know the 
interval limits, and it is an ad hoc assumption that the chaotic neighborhood is 
rectangular (this is a false assumption). Suppose that p is the smallest M-bit binary 
fraction such that p > 0.3 — p, and that p is the largest M-bit binary fraction such 
that p < 0.3 + A/J. In this case the number of M-bit fractions from 0.3 — A/3 to 
0.3 + AP is about 2 M - P) + 1. Thus 



2 M (P - p + 1) « 2 M [(0.3 + AP) - (0.3 - A/3)] + 1 = 2 M+1 AP + 1 = K p 



and by similar reasoning for a 



K a = 2 M+l Aa+\, 



(7.94a) 



(7.94b) 
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Figure 7.10 Transmitted (a) and reconstructed (b) message sequences. Here, the conditions 
of Fig. 7.9 hold except that the receiver uses a mismatched key a — 1.400001, fi = 0.300001. 
Thus, the message is eventually lost. (Note that the first few message samples seem to be 
recovered accurately.). 



which implies that the key size is (very approximately) 

K — K a Kp. 



(7.95) 



Even if Ace and A/3 are small, and the structure of the chaotic neighborhood is 
not rectangular, we can make M big enough (in principle) to generate a big key 
space. Apparently, Ruelle [21, p. 19] has shown that the Henon map is periodic 
(i.e., not chaotic) for a — 1.3, ^ = 0.3, so the size of the chaotic region is not very 
big. It is irregular and "full of holes" (the "holes" are key parameters that don't 
give chaotic outputs). In any case it seems that a large key size is possible. But as 
cautioned earlier, this is no guarantee of security. 

What does the transmitter send over the channel? This is the same as asking 
what the eavesdropper knows. From the encryption algorithm the eavesdropper 
knows the synchronizing elements yo,y\, and the cyphertext. The eavesdropper 
also knows the algorithm, but not the key. Is this enough to find the key? It is now 
obvious to ask if knowledge of yo, y\ gives the key away. This question may be 
easily (?) resolved as follows. 

From (7.84) and (7.86) for n e N, we have 



y n +\ = i -uy n + Pyn-i- 



(7.96) 
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Figure 7.11 Transmitted (a) and reconstructed (b) message sequences. Here, the conditions 
of Fig. 7.9 hold except the receiver and the transmitter do not perform arithmetic in identical 
fashion. In this case the order of two operations was reversed at the receiver. Thus, the 
message is eventually lost. (Note again that the first few message samples seem to be 
recovered accurately.). 



The encryption algorithm generates yo, y\, , , , , yti+i, which, from (7.96), must 
lead to the key satisfying the linear system of equations 



n 



y 2 N -i 
yl 



-yo 
-y\ 

-y>N-2 
-yN-i 



a 
(i 



-yi 
-yi 

- yN 
yN+\ 



(7.97) 



Compactly, we have Ya — y. This is an overdetermined system. The eavesdropper 
has only yo, y\, so from (7.97) the eavesdropper needs to solve 



y\ 



-yo 
-yi 



a 




yi 

>'3 



(7.98) 
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But the eavesdropper does not know y>2, yi as these were not transmitted. These 
elements become mixed with the message, and so are not available to the eaves- 
dropper. We immediately conclude that deadbeat synchronization is secure. 3 There 
is an alternative synchronization scheme [18] that may be proved to be insecure 
by this method of analysis. 

Does the cyphertext give key information away? This seems not to have a 
complete answer either. However, we may demonstrate that the system of D'Angeli 
et al. [18] (using deadbeat synchronization) is vulnerable to a known plaintext 4 
attack. Such a vulnerability is ordinarily sufficient to preclude using a method in 
a high-security application. The analysis assumes partial prior knowledge of the 
message (s n ). Let us specifically assume that 



$n 



E 

k=0 



a^n 



(7.99) 



but we do not know a^ or p. That is, the structure of our message is a polyno- 
mial sequence, but we do not know more than this. Combining (7.90) with (7.96) 
gives us 



1 



-2Sn-2- 



(7.100) 



It is not difficult to confirm that 



J «-i 



2/> 

= E 

k=0 



so that (7.100) becomes 



E aia i 



i+j=k 



(n - If 



(7.101) 



E 

k=0 



cikn k c n 



2p 
k=Q 



yfc\_ 



P 

^ Pak(n 

k=0 



T) k c n 



-2 = 1. 



(7.102) 



Remembering that the eavesdropper has cyphertext (c„) (obtained by eavesdrop- 
ping), the eavesdropper can use (7.102) to set up a linear system of equations in 
the key and message parameters <%. The code is therefore easily broken in this 
case. 

If the message has a more complex structure, (7.102) will generally be replaced 
by some hard-to-solve nonlinear problem wherein the methods of previous sections 
(e.g., Newton-Raphson) can be used to break the code. We conclude that, in spite 
of technical problems from the eavesdropper's point of view (ill conditioning, 
incomplete cyphertext sequences, etc.), the scheme in Ref. 18 is not secure. 

This conclusion assumes there are no other distinct ways to work with the encryption algorithm 
equations in such a manner as to give an equation that an eavesdropper can solve for the key knowing 
only yo. >'i, and the cyphertext. 

The message is also called plaintext. 
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PROBLEMS 

7.1. For the functions and starting intervals below solve fix) — using the 
bisection method. Use stopping criterion (7.13d) with e — 0.005. Do the 
calculations with a pocket calculator. 

(a) f(x) = log,* + 2x + 1, [a , b Q ] = [0.2, 0.3]. 

(b) f(x) = x 3 - cosx, [a , b ] = [0.8, 1.0]. 

(c) /(*)=x-e-*/ 5 ,[fl ,6o] = [!,l]. 

(d) fix)=x 6 -x-l,[ao,b ] = [l,l]. 

(e) /(*) = SS£ + exp(-x), [ fl0 , &o] = [3, 4]. 

(f) /(*) = SSi - , + 1, [« , fc ] = [1, 2]. 

7.2. Consider /(x) = sin(x)/x. This function has a minimum value for some x e 
[jr, 2jt]. Use the bisection method to find this x. Use the stopping criterion 
in (7.13d) with e — 0.005. Use a pocket calculator to do the computations. 

7.3. This problem introduces the variation on the bisection method called regula 
falsi (or the method of false position). Suppose that [oq, bo] brackets the root 
p [i.e., fip) = 0]. Thus, /(flo)/(^o) < 0. The first estimate of the root p, 
denoted by po, is where the line joining the points (ao, fi^o)) and (£>o, fiba)) 
crosses the x axis. 

(a) Show that 

h) - «o , , , 

Po = ao- -—— — — fia ). 

fibo) - fia ) 

(b) Using stopping criterion (7.13d), write pseudocode for the method of 
false position. 

7.4. In certain signal detection problems (e.g., radar or sonar) the probability of 
false alarm (FA) (i.e., of saying that a certain signal is present in the data 
when it actually is not) is given by 

r°° i 

Pp A = / -if-'c" 1 ' 2 ^, (7.P.1) 

J, Vip/2)2P/2 

where n is called the detection threshold. If p is an even number, it can be 
shown that (7.P.1) reduces to the finite series 

, ( " /2) " 1 1 /mt 

**-•-*' £ f I) • 

k=Q 

The detection threshold r\ is a very important design parameter in signal 
detectors. Often it is desired to specify an acceptable value for Pfa (where 
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< Pfa < 1), and then it is necessary to solve nonlinear equation (7.P.2) 
for r\. Let p — 6. Use the bisection method to find r\ for 

(a) P m = 0.001 

(b) Pfa = 0.01 

(c) P F a = 0.1 

7.5. We wish to solve 

f(x) = x 4 - 5 -x 3 + 5 -x - 1 = 
using a fixed-point method. This requires finding gix) such that 

g(x) -x — f{x). 

Find four different functions gix). 

7.6. Can the fixed-point method be used to find the solution to 

fix) = x 6 -x- 1 =0 

for the root located in the interval [1, |]? Explain. 

7.7. Consider the nonlinear equation 



fix) =x-e 



-x/5 



(which has a solution on interval [|, 1]). Use (7.32) to estimate a. Recalling 
that x n = g"xo (in the fixed-point method), if xq — 1, then use (7.22a) to 
estimate n so that dix n , x) < 0.005 [x is the root of fix) — 0]. Use a pocket 
calculator to compute x\, . . . , x n . 

7.8. Consider the nonlinear equation 

fix) =x-l- \e~ x = 

(which has a solution on interval [1, 1.2]). Use (7.32) to estimate a. Recalling 
that x n = g n XQ (in the fixed-point method), if xo = 1 then use (7.22a) to 
estimate n so that dix n , x) < 0.001 [x is the root of fix) — 0]. Use a pocket 
calculator to compute x\, . . . , x n . 

7.9. Problem 5.14 (in Chapter 5) mentioned the fact that orthogonal polynomi- 
als possess the "interleaving of zeros" property. Use this property and the 
bisection method to derive an algorithm to find the zeros of all Legendre poly- 
nomials P„ix) for n = 1, 2, . . . , N. Express the algorithm in pseudocode. Be 
fairly detailed about this. 
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7.10. We wish to find all of the roots of 

f{x) = x 3 - 3x 2 + Ax - 2 = 0. 

There is one real-valued root, and two complex-valued roots. It is easy to 
confirm that f(l) — 0, but use 

x 3 + 3x 2 +x + 2 
8(X) = 2^T1 

to estimate the real root p using fixed-point iteration [i.e, p n +\ — g(p n )]- 
Using a pocket calculator, compute only p\, p%, pj,, and p\, and use the 
starting point po — 2. Also, use the Newton-Raphson method to estimate 
the real root. Again choose po — 2, and compute only p\, pi, pj, and p\. 
Once the real root is found, finding the complex-valued roots is easy. Find 
the complex-valued roots by making use of the formula for the roots of a 
quadratic equation. 

7.11. Consider Eq. (7.36). Via Theorem 3.3, there is an a n between root p (f(p) = 
0) and the iterate p n such that 

f(p) ~ f(Pn) = f (1) (cc„)(p - Pn ). 

(a) Show that 

' fiPn) f m ( P n-l)\ 



Pn = (Pn ~ Pn-l) 



f(pn-i) / (1) (««) 



[Hint: Use the identity 1 = 



f(Pn-l) f W (Pn-l) I 
f (Pn-l) fW (Pn-l) '* 

(b) Argue that if convergence is occurring, we have 



lim \A„\ = 1. 

n— >oo 



Hence lim„ 



P-Pn 



Pn-pn-1 ) 

(c) An alternative stopping criterion for the Newton-Raphson method is to 
stop iterating when 

\f(Pn)\ + \Pn ~ Pn-\\ < <? 

for some suitably small e > 0. Is this criterion preferable to (7.42)? 
Explain. 
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7.12. For the functions listed below, and for the stated starting value po, use 
the Newton-Raphson method to solve f(p) — 0. Use the stopping crite- 
rion (7.42a) with e — 0.001. Perform all calculations using only a pocket 
calculator. 

(a) f(x) — x + tanx, po — 2. 

(b) f(x) = x 6 -x-l,p = l.5. 

(c) f(x) — x 3 — cosx, po — 1. 

(d) f(x) = x - e~ x l\ p = 1. 

7.13. Use the Newton-Raphson method to find the real-valued root of the polyno- 
mial equation 

1 1,1, 

/ (x) = 1 + -x + -x 2 + —x 3 = 0. 
J 2 6 24 

Choose starting point po — —2. Iterate 4 times. [Comment: Polynomial /(x) 
arises in the stability analysis of a numerical method for solving ordinary 
differential equations. This will be seen in Chapter 10, Eq. (10.83).] 

7.14. Write a MATLAB function to solve Problem 7.4 using the Newton-Raphson 
method. Use the stopping criterion (7.42a). 

7.15. Prove the following theorem (Newton-Raphson error formula). Let f(x) e 
C 2 [a, b], and f(p) = for some p e [a, b]. For p n e [a, b] with 

f(Pn) 
Pn + l -Pn f(l)(pn y 

there is a % n between p and p n such that 

1 2 / (2) (f«) 
P ~ Pn+1 = -~(P ~ Pn) -77777 ". 

2 f W (Pn) 

[Hint: Consider the Taylor series expansion of /(x) about the point x = p n 

fix) = f(p„) + (X- p„)f m ( Pn ) + \(X - Pn ) 2 f (2) ^n), 

and then set x = p.] 

7.16. This problem is about two different methods to compute */x. To begin, recall 
that if x is a binary floating-point number (Chapter 2), then it has the form 

x = xo.xi • • • x t x 2 e , 

where, since x > 0, we have xo = 0, and because of normalization x\ = 1. 
Generally, x^ e {0, 1}, and e is the exponent. If e = 2k (i.e., the exponent 
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is even), we do not adjust x. But if e = 2k + 1 (i.e., the exponent is odd), 
we shift the mantissa to the right by one bit position so that e becomes 
e — 2k J r 2. Thus, x now has the form 

x — a x 2 e , 

where a € [\, 1) in general, and e is an even number. Immediately, yfx — 
*fa x 2 e l 2 . From this description we see that any square root algorithm need 
work with arguments only on the interval [^,1] without loss of generality 
(w.l.o.g.). 



(a) Finding the square root of x — a is the same problem as solving f(x) — 
x 2 — a — 0. Show that 
the iterative algorithm 



x 2 — a — 0. Show that the Newton-Raphson method for doing so yields 



Pn+i = -( Pn + —\, (7.P.3) 

where p n — > */a. [Comment: It can be shown via an argument based 
on the theorem in Problem 7.15 that to ensure convergence, we should 
set po — I (2a + 1). A simpler choice for the starting value is p$ = 



i(i + 1) = | (since we know a € [4, 1]). 



3' 
2M i •-> — 8 V"" v " "* ^""" " = L 4 > 

(b) Mikami et al. (1992) suggest an alternative algorithm to find */x. They 
recommend that for a e [j, 1] the square root of a be obtained by the 
algorithm 

p n+l =/3(a- p 2 ) + p n . (7.P.4) 

Define error sequence (e n ) as e n — «Ja — p n . Show that 
e„+i = Pel + (1 - 2fi^/a)e„. 

[Comment: Mikami et al. recommend that po — 0.666667a + 0.354167.] 

(c) What condition on fi gives quadratic convergence for the algorithm in 
(7.P.4)? 

(d) Some microprocessors are intended for applications in high-speed digital 
signal processing. As such, they tend to be fixed-point machines with a 
high-speed hardware multiplier, but no divider unit. Floating-point arith- 
metic tends to be avoided in this application context. In view of this, 
what advantage might (7.P.4) have over (7. P. 3) as a means to compute 
square roots? 

7.17. Review Problem 7.10. Use x — xq + jx\ (xo, x\ e R) to rewrite the equation 
f(x) = jt 3 — 3x 2 + 4x — 2 — in the form 

fo(xo, *i) = 0, fi(xo,xi) = 0. 
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Use the Newton-Raphson method (as implemented in MATLAB) to solve 
this nonlinear system of equations for the complex roots of f(x) — 0. Use the 
starting points Jo = [ xo,o *o,i ] T — [ 2 2 ] T and 1q — [ 2 —2 ] T . 
Output six iterations in both cases. 

7.18. Consider the nonlinear system of equations 

f( X ,y) = X 2 + y 2 -l=0, 
g(x,y)=\x 2 + 4y 2 -l=0. 

(a) Sketch / and g on the (x, y) plane. 

(b) Solve for the points of intersection of the two curves in (a) by hand 
calculation. 

(c) Write a MATLAB function that uses the Newton-Raphson method to 
solve for the points of intersection. Use the starting vectors [xoyo] r = 
[±1 ± l] r . Output six iterations in all four cases. 

7.19. If y n+l = 1 - byl and x„ = Ux - ±) y„ + \ for n e Z+, then, if b = 
jk 2 — jX, show that 

%n + \ == XXn\i X n ). 

7.20. De Angeli, et al, [18] suggest an alternative synchronization scheme (i.e., 
alternative to deadbeat synchronization). This works as follows. Suppose 
that at the transmitter 

XQ,n + l = 1 -a*o,n + x l,«- 
*l,n+l = Pxo,„, 
y n = \-ax 2 n . 

The expression for y„ here replaces that in (7.86). At the receiver 

XQ, n +l — xi,„ + y„ 
X\,n+l = &XQ,n- 

(a) Error sequences are defined as 

0Xi,n == X[ n X{ n 

for i e {0, 1}. Find conditions on a and ft, giving linin^oo ||5x„|| = 
(Sx„ = [Sxo.n <5^i, n ] r ). 

(b) Prove that using y n = 1 — axh to synchronize the receiver and trans- 
mitter is not a secure synchronization method [i.e., an eavesdropper may 
collect enough elements from (y n ) to solve for the key {a, fi}]. 
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7.21. The chaotic encryption scheme of De Angeli et al. [18], which employs 
deadbeat synchronization, was shown to be vulnerable to a known plaintext 
attack, assuming a polynomial message sequence (recall Section 7.6). Show 
that it is vulnerable to a known plaintext attack when the message sequence 
is given by 

s n — a sin(o)M + </>) . 

Assuming that the eavesdropper already knows w and </>, show that a, a and 
jS can be obtained by solving a third-order linear system of equations. How 
many cyphertext elements c n are needed to solve the system? 

7.22. Write, compile, and run a C program to implement the De Angeli et al. 
chaotic encryption/decryption scheme in Section 7.6. (C is suggested here, 
as I am not certain that MATLAB can do the job so easily.) Implement both 
the encryption and decryption algorithms in the same program. Keep the 
program structure simple and direct (i.e., avoid complicated data structures, 
and difficult pointer operations). The program input is plaintext from a file, 
and the output is decrypted cyphertext (also known as "recovered plaintext") 
that is written to another file. The user is to input the encryption key {a, /3}, 
and the decryption key {a\, fi{\ at the terminal. Test your program out on 
some plaintext file of your own making. It should include keyboard charac- 
ters other than letters of the alphabet and numbers. Of course, your program 
must convert character data into a floating-point format. The floating-point 
numbers are input to the encryption algorithm. Algorithm output is also a 
sequence of floating-point numbers. These are decrypted using the decryp- 
tion algorithm, and the resulting floating-point numbers are converted back 
into characters. There is a complication involved in decryption. Recall that 
nominally the plaintext is recovered according to 

_ y«+2 

s n — 

Cn 

for n = 0, 1, . .., N — 1. However, this is a floating-point operation that 
incurs rounding error. It is therefore necessary to implement 

yn+2 -_ 

s„ = h offset 



for which offset is a small positive value (e.g., 0.0001). The rationale is 
as follows. Suppose that nominally (i.e., in the absence of roundoff) s n — 
yn+i/cn = 113.000000. The number 113 is the ASCII (American Standard 
Code for Information Interchange) code for some text character. Rounding in 
division may instead yield the value 112.999999. The operation of converting 
this number to an integer type will give 112 instead of 1 13. Clearly, the offset 
cures this problem. A rounding error that gives, for instance, 113.001000 is 
harmless. We mention that if plaintext (s„) is from sampled voice or video, 
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then these rounding issues are normally irrelevant. Try using the following 
key sets: 

a= 1.399, £ = 0.305, 

ai = 1.399, Pi =0.305, 

a= 1.399,0 = 0.305, 

ai = 1.389, Pi =0.304. 

Of course, perfect recovery is expected for the first set, but not the second set. 
[Comment: It is to be emphasized that in practice keys must be chosen that 
cause the Henon map to be chaotic to good approximation on the computer. 
Keys must not lead to an unstable system, a system that converges to a 
fixed point, or to one with a short-period oscillation. Key choices leading to 
instability will cause floating-point overflow (leading to a crash), while the 
other undesirable choices likely generate security hazards. Ideally (and more 
practically) the cyphertext array (floating-point numbers) should be written to 
a file by an encryption program. A separate decryption program would then 
read in the cyphertext from the file and decrypt it, producing the recovered 
plaintext, which would be written to another file (i.e., three separate files: 
original file of input plaintext, cyphertext file, and recovered plaintext file).] 
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Unconstrained Optimization 



8.1 INTRODUCTION 

In engineering design it is frequently the case that an optimal design is sought 
with respect to some performance criterion or criteria. Problems of this class are 
generally referred to as optimal design problems, or mathematical programming 
problems. Frequently nonlinearity is involved, and so the term nonlinear program- 
ming also appears. There are many subcategories of problem types within this broad 
category, and so space limitations will restrict us to an introductory presentation 
mainly of so-called unconstrained problems. Even so, only a very few ideas from 
within this category will be considered. However, some of the methods treated in 
earlier chapters were actually examples of optimal design methods within this cate- 
gory. This would include least-squares ideas, for example. In fact, an understanding 
of least-squares ideas from the previous chapters helps a lot in understanding the 
present one. Although the emphasis here is on unconstrained optimization problems, 
some consideration of constrained problems appears in Section 8.5. 



8.2 PROBLEM STATEMENT AND PRELIMINARIES 

In this chapter we consider the problem 

min/(;c) (8.1) 

jcsR" 

for which f(x) e R. This notation means that we wish to find a vector x that min- 
imizes the function f(x). We shall follow previous notational practices, so here 
x — [xq x\ ... x„_2 x n -\\ T . In what follows we shall usually assume that f(x) 
possesses all first- and second-order partial derivatives (with respect to the elements 
of x), and that these are continuous functions, too. In the present context the func- 
tion f(x) is often called the objective function. For example, in least-squares prob- 
lems (recall Chapter 4) we sought to minimize V(a) — \\f(x) — J2k=o a k<Pk(x)\\ 2 
with respect to a € R N . Thus, V(a) is an objective function. The least-squares 
problem had sufficient structure so that a simple solution was arrived at; specifi- 
cally, to find the optimal vector a (denoted a), all we had to do was solve a linear 
system of equations. 
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Now we are interested in solving more general problems. For example, we might 
wish to find x — [xqx\] t to minimize Rosenbrock' s function 



f(x) = l00(x 1 - X 2) 2 + (l-x Q ) 2 . 



(8.2) 



This is a famous standard test function (taken here from Fletcher [1]) often used 
by those who design optimization algorithms to test out their theories. A contour 
plot of this function appears in Fig. 8.1. It turns out that this function has a unique 
minimum at x = x = [1 l] r . As before, we have used a "hat" symbol to indicate 
the optimal solution. 

Some ideas from vector calculus are essential to understanding nonlinear opti- 
mization problems. Of particular importance is the gradient operator. 



3 3 3 

3xo 9*i 9x„_ 

For example, if we apply this operator to Rosenbrock' s function, then 



(8.3) 



V/(x) 



3*0 

df(x) 

dxi 



-400x (xi - x%) - 2(1 - x ) 



200(xi - x$ 



(8.4) 




Figure 8.1 Contour plot of Rosenbrock's function. 
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We observe that V/(Jc) = [0 0] T (recall that x — [1 l] T ). In other words, the 
gradient of the objective function at x is zero. Intuitively we expect that if x is 
to minimize the objective function, then we should always have V/(jc) = 0. This 
turns out to be a necessary but not sufficient condition for a minimum. To see this, 
consider g(x) — —x 2 (x e R) for which 

Vg(x) = -2x. 

Clearly, Vg(0) = yet x — maximizes g{x) instead of minimizing it. Consider 
h(x) — x 3 , so that 

Vh(x) = 3x 2 

for which Vft(0) = 0, but x — neither minimizes nor maximizes h(x). In this 
case x — corresponds to a one-dimensional version of a saddle point. 

Thus, finding x such that V/(x) = generally gives us minima, maxima, or 
saddle points. These are collectively referred to as stationary points. We need (if 
possible) a condition that tells us whether a given stationary point corresponds to 
a minimum. Before considering this matter, we note that another problem is that 
a given objective function will often have several local minima. This is illustrated 
in Fig. 8.2. The function is a quartic polynomial for which there are two values of 
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Figure 8.2 A function f(x) = x — 3.1x + 2x + 1 with two local minima. The local 
minimizers are near x = —0.5 and x = 2.25. The minimum near x = 2.25 is the unique 
global minimum of fix). 
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x such that V fix) — 0. We might denote these by x a and Xb- Suppose that x a is the 
local minimizer near x — —0.5 and Xb is the local minimizer near x — 2.25. We see 
that f(x a + S) > f(x a ) for all sufficiently small (but nonzero) S e R. Similarly, 
f(xb + S) > f(xb) for all sufficiently small (but nonzero) S e R. But we see that 
f(xb) < f(x a ), so Xb is the unique global minimizer for fix). An algorithm for 
minimizing f(x) should seek Xb\ that is, in general, we seek the global minimizer 
(assuming that it is unique). Except in special situations, an optimization algorithm 
is usually not guaranteed to do better than find a local minimizer, and is not 
guaranteed to find a global minimizer. Note that it is entirely possible for the 
global minimizer not to be unique. A simple example is f(x) — sin(x) for which 
the stationary points are x — (2k + 1)tt/2 (A: € Z). The minimizers that are a subset 
of these points all give —1 for the value of sin(x). 

To determine whether a stationary point is a local minimizer, it is useful to have 
the Hessian matrix 



V 2 f(x) 



d 2 f(x) 
dxt dxj 



eR" 



(8.5) 



!,;=0,l,...,n-l 



This matrix should not be confused with the Jacobian matrix (seen in Section 7.5.2). 
The Jacobian and Hessian matrices are not the same. The veracity of this may be 
seen by comparing their definitions. For the special case of n = 2, we have the 
2x2 Hessian matrix 



V 2 /(*) = 



d\f(x) d 2 f(x) 

3xq 3^o3xi 

S 2 f(x) d 2 f(x) 

3xi 3xQ dx 2 



(8.6) 



If f(x) in (8.6) is Rosenbrock's function, then (8.6) becomes 



V 2 /(*) = 



1200x2 - 400xi + 2 -400xo 
-400xo 200 



(8.7) 



The Hessian matrix for f(x) helps in the following manner. Suppose that x is 
a local minimizer for objective function f(x). We may Taylor-series-expand f(x) 
around x according to 



f{x + h) = f{x)+h'Vftx) 



\h T [V 2 f{x)]h 



(8.8) 



where h eR". (This will not be formally justified.) If f(x) is sufficiently smooth 
and \\h\\ is sufficiently small, then the terms in (8.8) that are not shown may 
be entirely ignored (i.e., we neglect higher-order terms). In other words, in the 
neighborhood of x 



fix + h)*> f(x) + h 1 V/(x) + W [V z fix)]h 



(8.9) 
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is assumed. But this is the now familiar quadratic form. For convenience we will 
(as did Fletcher [1]) define 

G(x) = V 2 f(x), (8.10) 

so (8.9) may be rewritten as 

f(x + h) % f(x) + h T Vf(x) +\h T G(x)h. (8.11) 

Sometimes we will write G instead of G(x) if there is no danger of confusion. 
In Chapter 4 we proved that (8.11) has a unique minimum iff G > 0. In words, 
f(x) looks like a positive definite quadratic form in the neighborhood of a local 
minimizer. Therefore, the Hessian of f(x) at a local minimizer will be positive 
definite and thus represents a way of testing a stationary point to see if it is a 
minimizer. (Recall the second-derivative test for the minimum of a single-variable 
function from elementary calculus, which is really just a special case of this more 
general test.) More formally, we therefore have the following theorem. 

Theorem 8.1: A sufficient condition for a local minimizer x is that V/(i) = 0, 
and G(x) > 0. 

This is a simplified statement of Fletcher's Theorem 2.1.1 [1, p. 14]. The proof 
is really just a more rigorous version of the Taylor series argument just given, and 
will therefore be omitted. For convenience we will also define the gradient vector 

gW = V/(x). (8.12) 

Sometimes we will write g for g(x) if there is no danger of confusion. 

If we recall Rosenbrock's function again, we may now test the claim made earlier 
that x = [1 l] T is a minimizer for f(x) in (8.2). For Rosenbrock's function the 
Hessian is in (8.7), and thus we have 



G(x) = 



802 -400 
-400 200 



(8.13) 



The eigenvalues of this matrix are A. — 0.3994, 1002. These are both bigger than 
zero so G(x) > 0. We have already remarked that g(x) — 0, so immediately from 
Theorem 8.1 we conclude that x — [1 l] T is a local minimizer for Rosenbrock's 
function. 



8.3 LINE SEARCHES 

In general, for objective function f(x) we wish to allow x e R"; that is, we seek the 
minimum of f(x) by performing a search in an n-dimensional vector space. How- 
ever, the one-dimensional problem is an important special case. In this section we 
begin by considering n — 1 , so x is a scalar. The problem of finding the minimum 
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of f(x) [where f(x) e R for all x e R] is sometimes called the univariate search. 
Various approaches exist for the solution of this problem, but we will consider only 
the golden ratio search method (sometimes also called the golden section search). 
We will then consider the backtracking line search [3] for the case where x e R" 
for any n > 1. 

Suppose that we know f(x) has a minimum over the interval [xf , xi]. Define 
I 3 — xi — xf , which is the length of this interval. The index j represents the jth 
iteration of the search algorithm, so it represents the current estimate of the interval 
that contains the minimum. Select two points xi and x J h (xi < xf) such that they 
are symmetrically placed in the interval [xf , xi]. Specifically 

xi-xj =xl-x{. (8.14) 

A new interval [xf ,x ] u ] is created according to the following procedure, such 
that for all j 

7^r = t>i. (8.i5) 

If f(x£) > f(xl) then the minimum lies in [xi, xi], and so 7 /+1 = xi — xi. Our 
new points are given according to 

x l — x a > x u — x m> x a = x b (8.16a) 



and 



xl +1 =xi + -I i+1 . (8.16b) 

T 



If f(xi) < f(xl), then the minimum lies in [xf , xf], and so / /+1 = x } b — xf . Our 
new points are given according to 



7+1 / 7+1 j ;+l _ J 



and 



xi, x^=x J a (8.17a) 



x J a +1 =xi-^I' +1 . (8.17b) 

Since 1° — x® — x® (j — indicates the initial case), we must have X® = x® — -1° 
and x® — xf + -1°. Figure 8.3 illustrates the search procedure. This is for the 
particular case (8.17). 

Because of (8.15), the rate of convergence to the minimum can be estimated. 
But we need to know t; see the following theorem. 

Theorem 8.2: The search interval lengths of the golden ratio search algorithm 
are related according to 

IJ = IJ+ 1 +iJ+ 2 (8.18a) 
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Figure 8.3 Illustrating the golden ratio search procedure. 

for which the golden ratio x is given by 

r = i(l + V5)« 1.62. 



(8.18b) 



Proof There are four cases to consider in establishing (8.18a). First, suppose 
that f(xi) > f(x J b ), so IJ +l = xi - x J a with 



J+ 1 



.7+1 



.7 + 1 



1 
-/ 

T 



/+] 



.7+1 



If it happens that f(x J a + ) > f{x[ ), then /'+ 2 = ^ 



/+i> 



^"+1 



.7+1 



Ki (via xi — x/ — x 3 u — xi). This implies that 7 ;+1 + / ;+2 = x^ — xi — V . 



On the other hand, if it happens that f(x 3 a ) < f(xi ), then 7 /+2 = x 



,7+1 



x /+1 - x /+1 



x ;+1 - x 1 



x, (via x, 
xi — xi). Again we have I J+i + I J+2 



7 + 1 



.7+1 _ 7+1 

A / — A w 

-x/ =/•'. 



.7+1 



Now suppose that f(x ] a ) < f(xi). Therefore, I J+l — x J h — xi with 



J+ 1 



.7+1 



If it happens that f(x J a )>f(xi + ), then V+ 2 = x 3 u 



-I j+l , 

x 



,7+1 



.7+1 



,7+1 



.7 + 1 



,7+1 



x i+l - x J ' - 
-7+1-, 



so that IJ +l + V+ 2 = x,; 



that /(x a ;+ ) < /(x^ + ), so therefore 7'+ 2 = x fc ;+ - x 



x; ; = /'. Finally, suppose 

,7+1 _ _; „7 _ v i „7 
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so yet again we conclude that /■ /+1 + L' +2 — xi — xj — P . Thus, (8.18a) 
now verified. 



Since 



— - I — = r and V = I j+1 + I j+2 , 



jj+l 77+2 



we have V +x = ±V and L' +1 = ±/ /+1 = XlJ, so 

V =IJ +1 + IJ +2 = -IJ + ±-I j , 



rl 



immediately implying that 



which yields (8.18b). 



T 2 = T + l, 



The golden ratio has a long and interesting history in art as well as science 
and engineering, and this is considered in Schroeder [2]. For example, a famous 
painting by Seurat contains figures that are proportioned according to this ratio 
(Fig. 5.2 in Schroeder [2] includes a sketch). 

The golden ratio search algorithm as presented so far assumes a user-provided 
starting interval. The golden ratio search algorithm has the same drawback as the 
bisection method for root finding (Chapter 7) in that the optimum solution must be 
bracketed before the algorithm can be successfully run (in general). On the other 
hand, an advantage of the golden ratio search method is that it does not need f(x) 
to be differentiable. But the method can be slow to converge, in which case an 
improved minimizer that also does not need derivatives can be found in Brent [4]. 

When setting up an optimization problem it is often advisable to look for ways 
to reduce the dimensionality of the problem, if at all possible. We illustrate this 
idea with an example that is similar in some ways to the least-squares problem 
considered in the beginning of Section 4.6. Suppose that signal fit) e R it e R) 
is modeled as 

f(t) = asm(—t + 4>)+ri(t), (8.19) 



where ??(f) is the term that accounts for noise, interference, or measurement errors 
in the data fit). In other words, our data are modeled as a sinusoid plus noise. 
The amplitude a, phase 4>, and period T are unknown parameters. We are assumed 
only to possess samples of the signal; thus, we have only the sequence elements 

fn = finT s ) = a sin fynT, + J + rjinT s ) (8.20) 

for n — 0, I, . . . , N — 1. The sampling period parameter T s is assumed to be 
known. As before, we may define an error sequence 

(In \ 

e n — /„ — a sin I — nT s + <f> I , (8.21) 
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where again n — 0, I, . . . , N — 1. The objective function for our problem (using a 
least-squares criterion) would therefore be 



N-l 

V{a,T,<t>)= J2 

n=0 



. lit 

f n -asm) — nT s 



-i2 



(8.22) 



This function depends on the unknown model parameters a, T, and <p in a very non- 
linear manner. We might formally define our parameter vector to be x — [a T 4>] T e 
R 3 . This would lead us to conclude that we must search (by some means) a three- 
dimensional space to find x to minimize (8.22). But in this special problem it is 
possible to reduce the dimensionality of the search space from three dimensions 
down to only one dimension. Reducing the problem in this way makes it solvable 
using the golden section search method that we have just considered. 
Recall the trigonometric identity 



sin(A + B) — sin A cos B + cos A sin B. 
Using this, we may write 



(2tt 
a sin — nl 



, lit \ (lit 

— a cosd) sin — nT s + a sin d> cos — nT s 



(8.23) 



(8.24) 



Define x — [xq x{\ and 



2tt \ (2tt 
sin ( — nT s I cos I — nT s 



-lT 



(8.25) 



We may rewrite e n in (8.21) using these vectors: 



e n — J n v n % ■ 



(8.26) 



Note that the approach used here is the same as that used to obtain e„ in Eqn. (4.99). 
Therefore, the reader will probably find it useful to review this material now. 
Continuing in this fashion, the error vector e — [eoei ■ ■ ■ epj-\] T , data vector / = 
[/b/i • • • /n-i] T , and matrix of basis vectors 



all may be used to write 



L "N-l J 



eR 



JVx2 



e — f — Ax, 



(8.27) 
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for which our objective function may be rewritten as 

V(x) = e T e = f T f - 2x T A T f + x T A T Ax, (8.28) 

which implicitly assumes that we already know the period T. If T were known 
in advance, we could use the method of Chapter 4 to minimize (8.28); that is, 
the optimum choice (least-squares sense) for x (denoted by x) satisfies the linear 
system 

A T Ax = A T f. (8.29) 

Because of (8.24) (with x — [xq x\] T ), we obtain 

xo = flcos(j!>, x\=asm<p. (8.30) 

Since we have x from the solution to (8.29), we may use (8.30) to solve for a 
and <p. 

However, our original problem specified that we do not know a, <p, or T in 
advance. So, how do we exploit these results to determine T as well as a and 4>1 
The approach is to change how we think about V (x) in (8.28). Instead of thinking 
of V(x) as a function of x, consider it to be a function of T, but with x — x as 
given by (8.29). From (8.25) we see that v n depends on T, so A is also a function 
of T, Thus, x is a function of T, too, because of (8.29). In fact, we may emphasize 
this by rewriting (8.29) as 

[A(T)] T A(T)x(T) = [A(T)ff. (8.31) 

In other words, A = A(T), andx = x (T). The objective function V(x) then becomes 

Vi(T) = V(x(T)) = f f - 2[x(T)] T A T (T)f + [x(T)f A T (T)A(T)x(T). 

(8.32) 
However, we may substitute (8.31) into (8.32) and simplify with the result that 

Vi(T) = f f - f A(T)[A T (T)A(T)r l A T (T)f. (8.33) 

We have reduced the search space from three dimensions down to one, so we 
may apply the golden section search algorithm (or some other univariate search 
procedure) to objective function V\(T). This would result in determining the opti- 
mum period T [which minimizes V\(T)]. We then compute A(T), and solve 
for x using (8.29) as before. Knowledge of x allows us to determine a and </> 
via (8.30). 

Figure 8.4 illustrates a typical noisy sinusoid and the corresponding objective 
function V\(T), In this case the parameters chosen were T = 24 h, T s — 5 min, 
a = 1, </> = — jt/10 radians, and N — 500. The noise component in the data was 
created using MATLAB's Gaussian random-number generator. The noise variance 
is a 2 — 0.5, and the mean value is zero. We observe that the minimum value of 
V\{T) certainly corresponds to a value of T that is at or close to 24 h. 
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Figure 8.4 An example of a noisy sinusoid (a) and the corresponding objective function 
Vi(T)(b). 



Appendix 8. A contains a sample MATLAB program implementation of the 
golden section search algorithm applied to the noisy sinusoid problem depicted 
in Fig. 8.4. In golden.m Topt is T , and eta rj is the random noise sequence r](nT s ). 

We will now consider the backtracking line search algorithm. The exposition 
will be similar to that in Boyd and Vandenberghe [5]. 

Now we assume that f(x) is our objective function with x e R". In a general 
line search we seek the minimizing vector sequence (x^'), k e Z + , and x w € R" 
(i.e., x^ = [xq x\ • • • x^^] 7 ) constructed according to the iterative process 



x ik+i) =x ik) + t m s (k) 



(8.34) 



where t^' e R + , and t w > except when x^ is optimal [i.e., minimizes f(x)]. 
The vector s^ is called the search direction, and scalar t ^ is called the step size. 
Because line searches (8.34) are descent methods, we have 



,(*+!) 



f(x^>) < f(x^>) 



(/<K 



(8.35) 



Ak) 



except when x ( ' is optimal. The "points" x 



(*+i) 



Ak) 



lie along a line in the direction 



in n -dimensional space R", and since the minimum of f(x) must lie in the 
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direction that satisfies (8.35), we must ensure that s^ satisfies [recall (8.12)] 

g(x (k) ) T s (k) = [Vf(x (k) )] T s (k) < 0. (8.36) 

Geometrically, the negative- gradient vector — g{x^) (which "points down") makes 
an acute angle (i.e., one of magnitude <90°) with the vector s", [Recall (4.130).] 
If s^ satisfies (8.36) for f(x^) (i.e., f(x) at x — x^), it is called a descent direc- 
tion for f(x) at x^\ A general descent algorithm has the following pseudocode 
description: 

Specify starting point x' 0) e R"; 

k:=0; 

while stopping criterion is not met do begin 

Find s^'; { determine descent direction } 

Find fW; { line search step } 

Compute x<*+ 1 > := x™ + t^s^ ■ 
{ update step } 

k:=k+-\; 

end ; 

Newton's method with the backtracking line search (Section 8.4) is a specific 
example of a descent method. There are others, but these will not be considered in 
this book. 

Now we need to say more about how to choose the step size t^ on the assump- 
tion that s w is known. How to determine the direction s'*' is the subject of 
Section 8.4. 

So far /|R" — > R. Subsequent considerations are simplified if we assume that 
f(x) satisfies the following definition. 

Definition 8.1: Convex Function Function /|R" -> R is called a convex 
function if for all $ e [0, 1], and for any x, y e R" 

f(0x + (1 - 6)y) < 9f(x) + (1 - $)f(y). (8.37) 

We emphasize that the domain of definition of f(x) is assumed to be R". It is 
possible to modify the definition to accommodate f(x) where the domain of f(x) 
is a proper subset of R". The geometric meaning of (8.37) is shown in Fig. 8.5 
for the case where x e R (i.e., one-dimensional case). We see that when f(x) 
is convex, the chord, which is the line segment joining (x, f(x)) to (y, f(y)), 
always lies above the graph of fix). Now further assume that f(x) is at least 
twice continuously differentiable in all elements of the vector x. We observe that 
for any x,y e R", if f(x) is convex, then 

f(y) > fix) + [Vf(x)] T (y - x). (8.38) 

From the considerations of Section 8.2 it is easy to believe that f(x) is convex iff 

V 2 f(x) > (8.39) 
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Figure 8.5 Graph of a convex function fix) (x e R) and the chord that connects the points 
(*,/(*)) and (y, /OO). 

for all x e R"; that is, fix) is convex iff its Hessian matrix is at least positive 
semidefinite (recall Definition 4.1). The function f(x) is said to be strongly convex 
iff V 2 /(x) > for all x e R". 

The backtracking line search attempts to approximately minimize f(x) along 
the line {x + ts\t > 0} for some given s (search direction at x). The pseudocode 
for this algorithm is as follows: 

Specify the search direction s; 

t :=1; 

while (f(x + fs) > f(x) + St[Wf(x)] T s) 

t := at; 

end ; 

In this algorithm < 8 < j, and < a < 1. Commonly, 8 e [0.01,0.30], and 
a e [0.1,0.5]. These parameter ranges will not be justified here. As suggested 
earlier, how to choose s will be the subject of the next section. The method is 
called "backtracking" as it begins with t = 1, and then reduces t by factor a until 
f(x + ts) < f{x) +8tV T f(x)s. [We have V T f(x) = [V/(x)] r .] Recall that s 
is a descent direction so that (8.36) holds, specifically, V r f(x)s < 0, and so if t 
is small enough [recall (8.9)], then 

f(x + ts) % fix) + tV T fix)s < fix) + 8tV T fix)s, 

which shows that the search must terminate eventually. We mention that the back- 
tracking line search will terminate even if fix) is only "locally convex" — convex 
on some proper subset S of R" . This will happen provided x e S in the algorithm. 



8.4 NEWTON'S METHOD 

Section 8.3 suggests attempting to reduce an n-dimensional search space to a 
one-dimensional search space. Of course, this approach seldom works, which is 
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why there is an elaborate body of methods on searching for minima in higher- 
dimensional spaces. However, these methods are too involved to consider in detail 
in this book, and so we will only partially elaborate on an idea from Section 8.2. 
The quadratic model from Section 8.2 suggests an approach often called New- 
ton's method. Suppose that x™ is the current estimate of the sought-after optimum 
x. Following (8.11), we have the Taylor approximation 

f(x (k) + S) % V(S) = f{x {k) ) + S T g(x (k) ) + ±8 T G(x (k) )8 (8.40) 

for which S e R" since x™ € R". Since x^ is not necessarily the minimum x, 
usually g(x^) ^ 0. Vector S is selected to minimize V(<5), and since this is a 
quadratic form, if G(x^) > 0, then 

G(x (k) )8 = -g(x (k) ). (8.41) 

The next estimate of x is given by 

X (k+D = x (k) + s _ (8 42) 

Pseudocode for Newton's method (in its most basic form) is 

Input starting point x* 0) ; 

k:=0; 

While stopping criterion is not met do begin 

G(xW)« := -g(x<Q); 

x (/c+1). = x W +a . 

/t:=/c+1; 

end; 

The algorithm will terminate (if all goes well) with the last vector x { • l ' as a good 
approximation to x. However, the Hessian G*- ' — G{x^') may not always be 
positive definite, in which case this method can be expected to fail. Modification 
of the method is required to guarantee that at least it will converge to a local 
minimum. Said modifications often involve changing the method to work with line 
searches (i.e., Section 8.3 ideas). We will now say more about this. 

As suggested in Ref. 5, we may combine the backtracking line search with the 
basic form of Newton's algorithm described above. A pseudocode description of 
the result is 

Input starting point x* 0) , and a tolerance e > 0; 

k := 0; 

S (0) := -[G(x (0) )n 1 g(x(°)); { search direction atx< 0) } 

X 2 := -gW; 

while k 2 > e do begin 

Use backtracking line search to find (^ forx^ and s^; 
X (fr+1) . = x (k) + t (k) s (k). 

S (*+D := -[G(x<' < + 1 >)r 1 g(x(' < + 1 >); 

A 2 :=-g 7 "(x<' ( + 1 ))s(' f + 1 ); 

k :=/c + 1; 

end; 
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The algorithm assumes that G(x^) > for all k. In this case [G(x^^)] _1 > as 
well. If we define (for all x e R") 

\\x\\ 2 G(y) ^x T [G(y)r 1 x, (8.43) 

then ||x||(j(y) satisfies the norm axioms (recall Definition 1.3), and is in fact an 
example of a weighted norm. But why do we consider X 2 as a stopping criterion in 
Newton's algorithm? If we recall the term jS T G(x^)S in (8.40), since S satisfies 
(8.41), at step k we must have 

\& T G(x^)& = i/(x«)[G(x«)]- 1 g(x«) = l 2\\g{x (k) )\\ 2 G(x(k)) = j^ 2 - (8.44) 

Estimate x w of x is likely to be good if (8.44) in particular is small [as opposed 
to merely considering squared unweighted norm g(x^) T g(x^)]. It is known that 
Newton's algorithm can converge rapidly (quadratic convergence). An analysis 
showing this appears in Boyd and Vandenberghe [5] but is omitted here. 

As it stands, Newton's method is computationally expensive since we must solve 
the linear system in (8.41) at every step k. This would normally involve applying 
the Cholesky factorization algorithm that was first mentioned (but not considered 
in detail) in Section 4.6. We remark in passing that the Cholesky algorithm will 
factorize G^ according to 

G (k) = LDL T , (8.45) 

where L is unit lower triangular and D is a diagonal matrix. We also mention that 
G^ 1 > iff the elements of D are all positive, so the Cholesky algorithm provides 
a built-in positive definiteness test. The decomposition in (8.45) is a variation on 
the LU decomposition of Chapter 4. Recall that we proved in Section 4.5 that 
positive definite matrices always possess such a factorization (see Theorems 4.1 
and 4.2). So Eq. (8.45) is consistent with this result. The necessity to solve a linear 
system of equations at every step makes us wonder if sensitivity to ill-conditioned 
matrices is a problem in Newton's method. It turns out that the method is often 
surprisingly resistant to ill conditioning (at least as reported in Ref. 5). 

Example 8.1 A typical run of Newton's algorithm with the backtracking line 
search as applied to the problem of minimizing Rosenbrock's function [Eq. (8.2)] 
yields the following output: 



k 


,(*) 


X 2 


x o 


(k) 
x\ 





1.0000 


800.00499 


2.0000 


2.0000 


1 


0.1250 


1.98757 


1.9975 


3.9900 


2 


1.0000 


0.41963 


1.8730 


3.4925 


3 


1.0000 


0.49663 


1.6602 


2.7110 


4 


0.5000 


0.38333 


1.5945 


2.5382 


5 


1.0000 


0.21071 


1.4349 


2.0313 
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k 


,(*) 


X 2 


x ik) 
x 


r (k) 

x l 


6 


0.5000 


0.14763 


1.3683 


1.8678 


7 


1.0000 


0.07134 


1.2707 


1.6031 


8 


1.0000 


0.03978 


1.1898 


1.4092 


9 


1.0000 


0.01899 


1.1076 


1.2201 


10 


1.0000 


0.00627 


1.0619 


1.1255 


11 


1.0000 


0.00121 


1.0183 


1.0350 


12 


1.0000 


0.00006 


1.0050 


1.0099 



The search parameters selected in this example are a — 0.5, 8 — 0.3, and e = 
.00001, and the final estimate is x (l3) — [1.0002 1.0003] 7 . For these same param- 
eters if instead x^ — [—1 l] T , then 18 iterations are needed, yielding x^ — 
[0.9999 0.9998] r . Figure 8.6 shows the sequence of points x w for the case 
x^ = [— 1 l] r . The dashed line shows the path from starting point [—1 l] T to 
the minimum at [1 l] r , and we see that the algorithm follows the "valley" to the 
optimum solution quite well. 




Figure 8.6 The sequence of points (x^ ') generated by Newton's method with the back- 
tracking line search as applied to Rosenbrock's function using the parameters a — 0.5, 
S = 0.3, and e = 0.00001 with *(°) = [-1 1] T (see Example 8.1). The path followed is 
shown by the dashed line. 
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8.5 EQUALITY CONSTRAINTS AND LAGRANGE MULTIPLIERS 

In this section we modify the original optimization problem in (8.1) according to 

min fix) 

*eR" , (8.46) 

subject to fi (jt) = for all i = 0, 1, . . . , m — 1 

where fix) is the objective function as before, and ft(x) = for i — 0, I, ... , 

m — 1 are the equality constraints. The functions /,- |R" -> R are equality constraint 

functions. The set F — {x\f{x) = 0, i = 0, . . . , m — 1} is called the feasible set. 

We are interested in 

f = f(x) = mm f(x). (8.47) 

xeF 

There may be more than one x — x e R" satisfying (8.47); that is, the set 

X = {x\x € F, f{x) = /} (8.48) 

may have more than one element in it. We assume that our problem yields X with 
at least one element in it (i.e., X ^ 0). 

Equation (8.47) is really a more compact statement of (8.46), and in words states 
that any minimizer x of fix) must also satisfy the equality constraints. We recall 
examples of this type of problem from Chapter 4 [e.g., the problem of deriving a 
computable expression for K2(A) and in the proof of Theorem 4.5]. More examples 
will be seen later. Generally, in engineering, constrained optimization problems are 
more common than unconstrained problems. However, it is important to understand 
that algorithms for unconstrained problems form the core of algorithms for solving 
constrained problems. 

We now wish to make some general statements about how to solve (8.46), and 
in so doing we introduce the concept of Lagrange multipliers. The arguments to 
follow are somewhat heuristic, and they follow those of Section 9.1 in Fletcher [1]. 

Suppose that x e R" is at least a local minimizer for objective function fix) e 
R. Analogously to (8.9), we have 

fix + 8)* fix) + gjix)& + ±8 T [V 2 fix)]8, (8.49) 

where 8 e R" is some incremental step away from x, and gtix) — V fix) e R" 
ii = 0, 1, . . . , m — 1) is the gradient of the ith constraint function at x. A feasible 
incremental step 8 must yield x + 8 e F, and so must satisfy 

fiix + 8) = Mx) = (8.50) 

for all i. From (8.49) this implies the condition that 8 must lie along feasible 
direction s e R" (at x — x) such that 

gfix)s = (8.51) 
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again for all ;'. [We shall suppose that the vectors gt(x) — V/(jc) are linearly 
independent for all /.] Recalling (8.36), if s were also a descent direction at x, then 

g T (x)s < (8.52) 

would hold (g(x) — V f{x) e R"). In this situation S would reduce f(x), as S is 
along direction s. But this is impossible since we have assumed that x is a local 
minimizer for f(x). [For any s at x, we expect to have g T (x)s = 0.] Consequently, 
no direction s can satisfy (8.51) and (8.52) simultaneously. This statement remains 
true if g(x) is a linear combination of g, (x), that is, if, for suitable I, € R we have 

m — 1 

g(x)=J^X i gi(x). 1 (8.53) 

i=0 

Thus, a necessary condition for x to be a local minimizer (or, more generally, a 
stationary point) of f(x) is that [rewriting (8.53)] 

m—\ 

g(x)-^2ii Si (x) = (8.54) 

i=0 

for suitable X,- € R which are called the Lagrange multipliers. We see that (8.54) 
can be expressed as [with V as in (8.3)] 



m— 1 



f(x)-J^kifi(x) 



= 0. (8.55) 



In other words, we replace the original problem (8.46) with the mathematically 
equivalent problem of minimizing 

m— 1 

L(x,X) = f(x)-J^Xifi(x), (8.56) 



Equation (8.53) may be more formally justified as follows. Note that the same argument will also 
extend to make (8.53) a necessary condition for x to be a local maximizer, or saddle point for f(x). 
Thus, (8.53) is really a necessary condition for x to be a stationary point of f(x). We employ proof by 
contradiction. Suppose that 

g(x) = Gk + h, 

where G = [go(x) ■ ■ ■ g m -\(x)] S R nxm , A, — [Xq ■ ■ ■ X m _ 1 ] T e R™ and h / 0. Further assume that 
h 6 R" is the component of g(x) that is orthogonal to all gi(x). Thus, G h = 0. In this instance 
s = -h will satisfy both (8.51) and (8.52) [i.e., g 1 ' (x)s = -[i T G T + h T ]h = -h T h < 0]. Satisfaction 
of (8.52) implies that a step & in the direction s will reduce fix) [i.e., f(x + 8) < /(*)]. But this cannot 
be the case since x is a local minimizer of /(x). Consequently, h ^ is impossible, which establishes 
(8.53). 
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called the Lagrangian function (or Lagrangian), where, of course, x € R" and 
X e R m . Since L|R" x R m — > R, in order to satisfy (8.55), we must determine 
x , X such that 

VL{x,X) = 0, 



where now instead [of (8.3)] we have V = 



V,- 



such that 



(8.57) 



3 



-,T 



3x dx n -i_ 



V/ 



3 

dT 



dX„ 



(8.58) 



Now we see that a necessary condition for a stationary point of f(x) subject to 
our constraints is that x and X form a stationary point of the Lagrangian function. 
Of course, to resolve whether stationary point x is a minimizer requires additional 
information (e.g., the Hessian). Observe that V^L(x, X) — [-fo(x) — fi(x) ■ ■ ■ — 
f m -\(x)] T , so V\L(x, X) — implies that ft(x) — for all i. This is why we take 
derivatives of the Lagrangian with respect to all elements of A; it is equivalent to 
imposing the equality constraints as in the original problem (8.46). 

We now consider a few examples of the application of the method of Lagrange 
multipliers. 

Example 8.2 This example is from Fletcher [1, pp. 196-198]. Suppose that 

f(x) = xo + x\, fo(x) = x% - xi = 0, 

so x — [xq x\\ T e R 2 , and the Lagrangian is 

L(x, X) — xq + x\ — X(x Q — x\). 

Clearly, to obtain stationary points, we must solve 

dL 



dxo 


= 1 - 2Xx = 


dL 

3xi 


= 1+A. = 0, 


dL 
~8X 


= Xq — X\ — 0. 



Immediately, X — — 1, so that 1 — 2Xxq — yields xq = — i, and xi. — x\ =0 



yields x\ 



Thus 



-lT 



1 1 

~2 4 
Is x really a minimizer, or is it a maximizer, or saddle point? 
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An alternative means to solve our problem is to recognize that since Xq — x\ =0, 
we can actually minimize the new objective function 

/'(x ) = X + X\\ Xi=x 2 — X + x\ 

with respect to xo instead. Clearly, this is a positive definite quadratic with a well- 
defined and unique minimum at xo = — \. Again we conclude x = [— | \Y , and 
it must specifically be a minimizer. 

In Example 8.2, / = f(x) — xq + x\ — — j > — oo only because of the con- 
straint, /o(x) = x^ — x\ — 0. Without such a constraint, we would have / = — oo. 

We have described the method of Lagrange multipliers as being applied largely to 
minimization problems. But we have noted that this method applies to maximization 
problems as well because (8.57) is the necessary condition for a stationary point, 
and not just a minimizer. The next example is a simple maximization problem from 
geometry. 

Example 8.3 We wish to maximize 

F(x, y) = 4xy 

subject to the constraint 

C(x,y) = \x 2 + y 2 -\=0. 

This problem may be interpreted as the problem of maximizing the area of a 
rectangle of area Axy such that the corners of the rectangle are on an ellipse centered 
at the origin of the two-dimensional plane. The ellipse is the curve C(x, y) — 0. 
(Drawing a sketch is a useful exercise.) 

Following the Lagrange multiplier procedure, we construct the Lagrangian 
function 

G(x, y, X) = Axy + \{\x 2 + y 2 - 1) 

Taking the derivatives of G and setting them to zero yields 



(8.59a) 
(8.59b) 
(8.59c) 



dG 

9x 


1 

= 4v + -Ax = 

2 


dG 

~dy~ 


= 4x + 2Xy = 


dG 


= -x 2 + y 2 - 1 = 
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From (8.59a,b) we have 

v x 

A. = -8-, and k = -2-, 

x y 

which means that — 2x/y = — %y/x, or x 2 — Ay 2 . Thus, we may replace x 2 by Ay 2 
in (8.59c), giving 

2y 2 - 1 = 

for which y 2 — j, and so x 2 — 2. From these equations we easily obtain the loca- 
tions of the corners of the rectangle on the ellipse. The area of the rectangle is also 
seen to be four units. 

Example 8.4 Recall from Chapter 4 that we sought a method to determine 
(compute) the matrix 2-norm 

||A|| 2 = max ||Ax|| 2 . 

Chapter 4 considered only the special case x e R 2 (i.e., n — 2). Now we consider 
the general case for which n > 2. 

I 2 

our problem is to maximize x T Rx subject to the equality constraint ||x||2 = 1 (or 
equivalently x T x — 1). The Lagrangian is 



Since ||Ax|| 2 = x T A T Ax = x T Rx with R = R T and R > (if A is full rank), 



L(x, X) = x T Rx - X(x T x - 1) 
since f(x) — x T Rx and fo(x) — x T x — 1. Now 

n—\ n — \ 

fix) = x T Rx = ^^XjXjrij 
i=0 y=0 

n — 1 n—\n—\ 

= £ 
1=0 



so that (using r,y = r ; ,) 



3/ 

- — = 2r kk x k 
ox k 



= 2r kk x k 



: ? r " + 2^2^ 


X[ X j f"ij 


i=0 ;=0 




«¥j 




n-l 


n-l 


X! ■ r ./^/ + 


£ xir ik 


7=0 


i =0 


/#* 


1 j=k 


n-l 


n-l 


Z! r v^ + 


J2 r kJ x J 


;=0 


.7 = 


/#* 


jl^k 
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This reduces to 

„j. n—\ n—\ 

— = 2r kk x k +2 ^ r kj Xj — 2 ^ r kj Xj 
,/=0 j=o 

for all k = 0, 1, . . . , n — 1. Consequently, V x f(x) — 2Rx. Similarly, V x fo(x) — 
2x. Also, V^L(x, 1) = — x T x + 1 = — fo(x), and so VL(x,A.) = yields the 
equations 

2Rx - 2Xx = 0, 
jc r x -1=0. 

The first equation states that the maximizing solution (if it exists) must satisfy the 
eigenproblem 

Rx — Xx 

for which X is an eigenvalue and x is the corresponding eigenvector. Consequently, 
x T Rx — Xx T x, so 



x T Rx \\Ax\\ 



2 

2 _ II „„,,2 



A=^^ = ^-^ = ||Ax|| 



11*11 



2 



must be chosen to be the biggest eigenvalue of R. Since R > 0, such an eigenvalue 
will exist. As before (Chapter 4), we conclude that 

IIAH^A.,,-! 

for which X n -\ is the largest of the eigenvalues A-o, . . . , X n -\ of R — A T A {X n -\ > 
K-2 > ■ ■ ■ > X > 0). 



APPENDIX 8.A MATLAB CODE FOR GOLDEN SECTION SEARCH 

% 

% SineFitl.m 

% 

% This routine computes the objective function V_1 (T) for user input 

% T and data vector f as required by the golden section search test 

% procedure golden. m. Note that Ts (sampling period) must be consistent 

% with Ts in golden. m. 

% 

function V1 = SineFitl (f ,T) ; 

N = length(f); % Number of samples collected 

Ts = 5*60; % 5 minute sampling period 
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% Compute the objective function V_1 (T) 

n = [0:N-1] ; 

T = T*60*60; 

A= [ sin(2*pi*Ts*n/T) . ' cos(2*pi*Ts*n/T) . ' ]; 

B = inv(A. '*A) ; 

V1 = f . '*f - f . ' *A*B*A. '*f ; 



% golden. m 

% This routine tests the golden section search procedure of Chapter 8 
% on the noisy sinusoid problem depicted in Fig. 8.4. 

% This routine creates a test signal and uses SineFitl.m to compute 

% the corresponding objective function V_1 (T) (given by Equation (8.33) 

% in Chapter 8) . 



function Topt = golden 

% Compute the test signal f 

N = 500; % Number of samples collected 

Ts = 5*60; % 5 minute sampling period 

T = 24*60*60; % 24 hr period for the sinusoid 
phi = -pi/10; % phase angle of sinusoid 
a = 1.0; % sinusoid amplitude 

var = .5000; % desired noise variance 

std = sqrt(var) ; 

eta = std*randn(1 ,N) ; 

for n = 1 :N 

f(n) = a*sin(((2*pi*(n-1)*Ts)/T) + phi); 

end; 
f = f + eta; 
f = f.'; 

% Specify a starting interval and initial parameters 
% (units of hours) 

xl = 16; 
xu = 32; 

tau = (sqrt(5)+1 ) /2; % The golden ratio 

tol = .05; % Accuracy of the location of the minimum 

% Apply the golden section search procedure 

I = xu - xl; % length of the starting interval 
xa = xu - I/tau; 
xb = xl + I/tau; 
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while I > tol 

if SineFitl (f ,xa) >= SineFitl (f ,xb) 

I = xu - xa; 

xl = xa; 

temp = xa; 

xa = xb; 

xb = temp + I/tau; 
else 

I = xb - xl; 

xu = xb; 

temp = xb; 

xb = xa; 

xa = temp - I/tau; 
end 
end 

Topt = (xl + xu)/2; % Estimate of optimum choice for T 
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PROBLEMS 

8.1. Suppose A e R" x ", and that A is not symmetric in general. Prove that 

V{x T Ax) = (A + A T )x, 

where V is the gradient operator [Eq. (8.3)]. 

8.2. This problem is about ideas from vector calculus useful in nonlinear opti- 
mization methods. 

(a) If s, x, x' e R", then a line in R" is defined by 

x — x(a) — x + as, a € R. 

Vector s may be interpreted as determining the direction of the line in the 
«-dimensional space R". The notation x(a) implies x(a) — [xo(a)x\(a) 
■ ■ ■ x n _\ (a)] T . Prove that the slope df/da of f(x(a)) e R along the line 
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at any x(a) is given by 

df T 
— =s T Vf, 
da 

where V is the gradient operator [see Eq. (8.3)]. (Hint: Use the chain 

rule for derivatives.) 

(b) Suppose that u(x), v(x) e R" (again x e R"). Prove that 

V{u T v) = (V/)v + (Vv T )u. 

8.3. Use the golden ratio search method to find the global minimizer of the 

polynomial objective function in Fig. 8.2. Do the computations using a pocket 

o 

l / 

8.4. Review Problem 7.4 (in Chapter 7). 



calculator, with starting interval [xf, x®] = [2, 2.5]. Iterate 5 times. 



Use a MATLAB implementation of the golden ratio search method to find 
detection threshold r\ for Pfa = 0.1,0.01, and 0.001. The objective func- 
tion is 



f(ri) 



(p/2)-l , , 

k=Q 



Make reasonable choices about starting intervals. 

8.5. Suppose x — [xq x\\ t e R 2 , and consider 

f( x ) = Xq + xoxi + (1 + x\) 2 . 

Find general expressions for the gradient and the Hessian of f(x). Is G(0) > 
0? What does this signify? Use the Newton-Raphson method (Chapter 7) 
to confirm that x — [0.6959 — 1.3479] r is a stationary point for f(x). 
Select Jo = [0.7000 - 1.3] r as the starting point. Is G(x) > 0? What does 
this signify? In this problem do all necessary computations using a pocket 
calculator. 

8.6. If x e R" is a local minimizer for f(x), then we know that G(x) > 0, that 
is, Gix) is positive definite (pd). On the other hand, if x is a local maximizer 
of f(x), then — G(x) > 0. In this case we say that G(x) is negative definite 
(nd). Show that 

f{x) = (XI - X\f + Xq 5 

has only one stationary point, and that it is neither a minimizer nor a maxi- 
mizer of f(x). 

8.7. Take note of the criterion for a maximizer in the previous problem. For both 
a — 6 and a — 8, find all stationary points of the function 

f(x) = 2xq — 3x — axoxi(xo — x\ — 1). 

Determine which are local minima, local maxima, or neither. 
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8.8. Write a MATLAB function to find the minimum of Rosenbrock's function 
using the Newton algorithm (basic form that does not employ a line search). 
Separate functions must be written to implement computation of both the 
gradient and the inverse of the Hessian. The function for computing the 
inverse of the Hessian must return an integer that indicates whether the Hes- 
sian is positive definite. The Newton algorithm must terminate if a Hessian 
is encountered that is not positive definite. The I/O is to be at the terminal 
only. The user must input the starting vector at the terminal, and the program 
must report the estimated minimum, or it must print an error message if the 
Hessian is not positive definite. Test your program out on the starting vectors 

x (0) = [_3 _3f an d x ®) = [0 10] r 

8.9. Write and test your own MATLAB routine (or routines) to verify Example 8.1. 

8.10. Find the points on the ellipse 

2 2 

x L y L 
a 2 b 2 

that are closest to, and farthest from, the origin (x, y) — (0, 0). Use the 
method of Lagrange multipliers. 

8.11. The theory of Lagrange multipliers in Section 8.5 is a bit oversimplified. 
Consider the following theorem. Suppose that x e R" gives an extremum 
(i.e., minimum or maximum) of f(x) e R among all x satisfying g(x) — 0. 
If /, g e C l [D] for a domain DcR containing x, then either 

g(x) = and Vg(x) = 0, (8.P.1) 

or there is a X e R such that 

g(x) = and V/(x) - XVg(x) = 0. (8.P.2) 

From this theorem, candidate points x for extrema of f(x) satisfying g(x) — 
therefore are 

(a) Points where / and g fail to have continuous partial derivatives. 

(b) Points satisfying (8.P.1). 

(c) Points satisfying (8. P. 2). 

In view of the theorem above and its consequences, find the minimum dis- 
tance from x = [xq x\ X2\ T — [0 — l] T to the surface 

g( X ) — Xq + x\ — x\ — 0. 
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8.12. A second-order finite-impulse response (FIR) digital filter has the frequency 
response H{e> 01 ) — ^k=Q n k e ~'' Mk ^ where h.% e R are the filter parameters. 
Since H(e J0} ) is 2tt — periodic we usually consider only w e [—it, it]. The 
DC response of the filter is H(l) — H(e>°) — J2k=o n k- Define the energy 
of the filter in the band [— a> p , a> P ] to be 



1 C m p , 

E=—\ \H(en\ 2 da>. 
2jt J-co„ 



Find the filter parameters h — [ho hi h-^ e R 3 such that for u> p — n/2 
energy E is minimized subject to the constraint that H(\) — 1 (i.e., the gain 
of the filter is unity at DC). Plot \H(e joJ )\ for co e [-n,n]. [Hint: E will 
have the form E — h T Rh e R, where R e R 3x3 is a symmetric Toeplitz 
matrix (recall Problem 4.20). Note also that \H(e^)\ 2 = //(e' a, )//*(e-'' w ).] 

8.13. This problem introduces incremental condition estimation (ICE) and is based 
on the paper C. H. Bischof, "Incremental Condition Estimation," SIAM J. 
Matrix Anal. Appl. 11, 312-322 (April 1990). ICE can be used to estimate 
the condition number of a lower triangular matrix as it is generated one 
row at a time. Many algorithms for linear system solution produce triangular 
matrix factorizations one row or one column at a time. Thus, ICE may be 
built into such algorithms to warn the user of possible inaccuracies in the 
solution due to ill conditioning. Let A n be an n x n matrix with singular 
values a\(A n ) > • • ■ > <r n (A n ) > 0. A condition number for A n is 

/, s °"i( A «) 

Gn(An) 

Consider the order n lower triangular linear system 

L n x n — d n . (8. P. 3) 

The minimum singular value, a n {L n ), of L n satisfies 

n w l|fif " 112 

On\E n ) < ~ — 

I 1*7! Il2 

(| \x„ I Ij = JZ( x n !')• Thus, an estimate (upper bound) of this singular value is 

1141b 



o n (L n ) = 



I 1*7! lb 



We would like to make this upper bound as small as possible. So, Bischof 
suggests finding x„ to satisfy (8.P.3) such that ||x„||2 is maximized subject 
to the constraint that ||<i n |b = 1. Given x n -\ such that L„_ix„_i = d n -\ 
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with |K_i|| 2 = 1 [which gives us cr„_i(L„_i) 
c n such that ||x„||2 is maximized where 



1/II^B-ilb], find s„ and 



L n -\ 

T 



V 







s n d n -\ 



d n , 



SnXn — 1 

(c n - s„a n )/y n 



Find a n ,c n ,s n . Assume a n ^0. (Comment: The indexing of the singular 
values used here is different from that in Chapter 4. The present notation is 
more convenient in the present context.) 

8.14. Assume that A e R mx " with m > n, but rank(A) < n is possible. As usual, 



\x\\i — x x. Solve the following problem: 



min 1 1 Ax 



Hi 



8\\x\\l 



where S > 0. This is often called the Tychonov regularization problem. It 
is a simple ploy to alleviate problems with ill-conditioning in least-squares 
applications. Since A is not necessarily of full rank, we have A T A > 0, but 
not necessarily A T A > 0. What is rank(A r A + 5/)? Of course, / is the order 
n identity matrix. 
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Numerical Integration 
and Differentiation 



9.1 INTRODUCTION 

We are interested in how to compute the integral 



r 

J a 



f(x)dx (9.1) 



for which f(x) e R (and, of course, x e R). Depending on f(x), and perhaps also 
on [a, b], the reader knows that "nice" closed-form expressions for / rarely exist. 
This forces us to consider numerical methods to approximate /. We have seen from 
Chapter 3 that one approach is to find a suitable series expansion for the integral 
in (9.1). For example, recall that we wished to compute the error function 

2 f x 2 
erf(x) = —= \ e~' dt, (9.2) 

V^ Jo 



which has no antiderivative (i.e., "nice" formula). Recall that the error function is 
crucial in solving various problems in applied probability that involve the Gaussian 
probability density function [i.e., the function in (3.101) of Chapter 3]. The Taylor 
series expansion of Eq. (3.108) was suggested as a means to approximately evaluate 
erf(x), and is known to be practically effective if x is not too big. If x is large, then 
the asymptotic expansion of Example 3.10 was suggested. The series expansion 
methodology may seem to solve our problem, but there are integrals for which it 
is not easy to find series expansions of any kind. 

A recursive approach may be attempted as an alternative. An example of this was 
seen in Chapter 5, where finding the norm of a Legendre polynomial required solv- 
ing a recursion involving variables that were certain integrals [recall Eq. (5.96)]. 
As another example of this approach, consider the following case from Forsythe 
et al. [1]. Suppose that we wish to compute 



In = f X n i 

JO 



"' dx (9.3) 
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for any n e N (natural numbers). Recalling integration by parts, we see that 



f x n e x - l dx=x n e x - x \l- f 
Jo Jo 



■ l 

/o Jo 

so 



nx" l e x ' dx = 1 — n£„_i, 



E n = l-nE n -i (9.4) 

for n = 2, 3, 4, It is easy to confirm that 



= 1 



xe* dx = — . 
e 



This is the initial condition for recursion (9.4). We observe that E n > for all 
n. But if, for example, MATLAB is used to compute £"19, we obtain computed 
solution £19 = —5.1930, which is clearly wrong. Why has this happened? 
Because of the need to quantize, E\ is actually stored in the computer as 

Ex = Ei + e 

for which e is some quantization error. Assuming that the operations in (9.4) do 
not lead to further errors (i.e., assuming no rounding errors), we may arrive at a 
formula for £„: 

E 2 =l-2E 1 = E 2 + (-2)6, 

£3 = 1 - 3£ 2 = £3 + (~3)(-2)e, 

£4 = 1 - 4£ 3 = £4 + (-4) (-3) (-2)6, 

and so on. In general 

£„ = £„ + (-l)"- 1 l-2-3---(/i- 1)m6, (9.5) 



£„-£„ = (- \) n - l n\€. (9.6) 

We see that even a very tiny quantization error 6 will grow very rapidly during 
the course of the computation even without any additional rounding errors at all ! 
Thus, (9.4) is a highly unstable numerical procedure, and so must be rejected as 
a method to compute (9.3). However, it is possible to arrive at a stable procedure 
by modifying (9.4). Now observe that 



£„ = / x "e x ~ 1 dx< / x n dx= . 

Jo Jo n+l 



(9.7) 
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implying that E n -> as n increases. Instead of (9.4), consider the recursion 
[obtained by rearranging (9.4)] 

£„_i = -(l-£„). (9.8) 

n 

If we wish to compute E m , we may assume E„ = for some n significantly bigger 
than m and apply (9.8). From (9.7) the error involved in approximating E n by zero 
is not bigger than l/(w + 1). Thus, an algorithm for E m is 

E k -i = -(1 - E k ) 
k 

for k = n,n — I, ... ,m + 2,m + I, where E n — 0. At each stage of this algorithm 
the initial error is reduced by factor \jk rather than being magnified as it was in 
the procedure of (9.4). 

Other than potential numerical stability problems, which may or may not be 
easy to solve, it is apparent that not all integration problems may be cast into a 
recursive form. It is also possible that f(x) is not known for all x e R. We might 
know f(x) only on a finite subset of R, or perhaps a countably infinite subset of 
R. This situation might arise in the context of obtaining f(x) experimentally. Such 
a scenario would rule out the previous suggestions. 

Thus, there is much room to consider alternative methodologies. In this chapter 
we consider what may be collectively called quadrature methods. For the most part, 
these are based on applying some of the interpolation ideas considered in Chapter 6. 
But Gaussian quadrature (see Section 9.4) also employs orthogonal polynomials 
(material from Chapter 5). 

This Chapter is dedicated mainly to the subject of numerical integration by 
quadratures. But the final section considers numerical approximations to deriva- 
tives (i.e., numerical differentiation). Numerical differentiation is relevant to the 
numerical solution of differential equations (to be considered in later chapters), 
and we have mentioned that it is relevant to spline interpolation (Section 6.5). In 
fact, it can also find a role in refined methods of numerical integration (Section 9.5). 



9.2 TRAPEZOIDAL RULE 

A simple approach to numerical integration is the following. 

In this book we implicitly assume all functions are Riemann integrable. From 
elementary calculus such integrals are obtained by the limiting process 



pb i n 

I b — a -r-^ 

= / f(x)dx= lim Tfixl) 



-b 

(9.9) 

k=\ 



for which xq — a,x n — b, and for which the value / is independent of the point xZ e 
[xjfc-i, Xfc]. We remark that not all functions f(x) satisfy this requirement and so, 
as was mentioned in Chapter 3, not all functions are Riemann integrable. However, 
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we will ignore this potential problem. We may approximate / according to 

b — a -r-^ 

/« — y,f(4)- ( 9 - 10 ) 

k=\ 

Such an approximation is called the rectangular rule (or rectangle rule) for numer- 
ical integration, and there are different variants depending on the choice for x\. 
Three possible choices are shown in Fig. 9.1. We mention that all variants involve 
assuming f(x) is piecewise constant on [xk-i, Xk], and so amount to the constant 
interpolation of f(x) (i.e., fitting a polynomial which is a constant to f(x) on 
some interval). Define 



h 



From Fig. 9.1, the right-point rule uses 



while the left-point rule uses 



■kh. 



x\ = a + (k — \)h, 



(9.11a) 



(9.11b) 




Figure 9.1 Illustration of the different forms of the rectangular rule: (a) right-point rule; 
(b) left-point rule; (c) midpoint rule. In all cases xq = a and x n = b. 
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and the midpoint rule uses 



x^ = a + (k — l)h, 



(9.11c) 



where in all cases k — 1, 2, . . . , n — \,n. The midpoint rule is often preferred 
among the three as it is usually more accurate. However, the rectangular rule is often 
(sometimes unfairly) regarded as too crude, and so the following (or something still 
"better") is chosen. 

It is often better to approximate f(x) with trapezoids as shown in Fig. 9.2. This 
results in the trapezoidal rule for numerical integration. From Fig. 9.2 we see that 
this rule is based on the linear interpolation of the function f(x) on [x k -\,x k ]. 
The approximation to f* k f(x) dx is given by the area of the trapezoid: this is 



f 

J XI 



f(x) dx % - [f(x k ) + f(x k -i)] (x k - x k -i) = - [f(x k -\) + f(x k )] h 



(9.12) 
[We have h — x k — x k -\ — (b — a)/n.] It is intuitively plausible that this method 
should be more accurate than the rectangular rule, and yet not require much, if 
any, additional computational effort to implement it. Applying (9.12) for k = 1 to 
k — n, we have 

fb h « 

/ f(x)dx*T(n) = -J^[f(x k -i) + f(x k )], (9.13) 

Ja Z k=\ 

and the summation expands out as 

Tin) = ^ [f(x ) + 2f(xi) + 2f(x 2 ) + ■■■ + 2f(x n -i) + f(x n )] , (9.14) 
where n e N (set of natural numbers). 




Figure 9.2 Illustration of the trapezoidal rule. 
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We may investigate the error behavior of the trapezoidal rule as follows. The 
process of analysis begins by assuming that n = 1; that is, we linearly interpolate 
fix) on [a, b] using 

p(x) = f(a)— b + fQ,)^-, (9.15) 

a — b b — a 

which is from the Lagrange interpolation formula [recall (6.9) for n — 1]. It must 
be the case that for suitable error function e(x), we have 

f(x) = p(x) + e(x) (9.16) 

with x e [a, b]. Consequently 



rb rb pb 

I f(x)dx— I p(x)dx+ I e(x) dx 

Ja Ja Ja 



= b —^U(.a) + f{b)] + f eix) dx = Til) + E T(1) , (9.17) 

where 

E T(l) = / eix)dx, (9.18) 

Ja 

which is the error involved in using the trapezoidal rule. Of course, we would like 
a suitable bound on this error. To obtain such a bound, we will assume that f^(x) 
and f^ix) both exist and are continuous on [a, b]. Let x be fixed at some value 
such that a < x < b, and define 

it — a)(t — b) 

git) = fit) - pit) - [fix) - pix)U ^ TT < 9 - 19 ) 

(x — a)ix — b) 

for t e [a, b]. It is not difficult to confirm that 

g(a) = g(b) = g(*) = 0, 

so g(f) vanishes at three different places on the interval [a, b]. Rolle's theorem 1 
says that there are points fi e (a, x), %2 € (x, £>) such that 

g (1) (ti) = g (1) fe) = o. 

Thus, g^(f) vanishes at two different places on ia,b), so yet again by Rolle's 
theorem there is a £ e (£i , £2) suc h that 

g (2) (f) = 0. 

This was proved by Bers [2], but the proof is actually rather lengthy, and so we omit it. 
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We note that f = £(x), that is, point f depends on x. Therefore 

g (2) (?) = / (2) (?) _ <2) (|) _ [/(x) _ (jt)] 2 = Q (9 2Q) 

(x — a)(x — D) 

The polynomial p(x) is of the first degree so // 2 ^(£) = 0. We may use this in 
(9.20) and rearrange the result so that 

fix) = p(x) + I/( 2 >(f (*))(* " «)(* " « (9.21) 

for any x e (a, b). This expression also happens to be valid at x — a and at x — b, 
so 

e(x) = f{x) - pix) = ±/( 2 >(t (*))(* - a)(* - &) (9.22) 

for x e [a,b]. But we need to evaluate £t(1) m (9.18). The second mean-value 
theorem for integrals states that if fix) is continuous and g(x) is integrable (Rie- 
mann) on [a, b], and further that gix) does not change sign on [a, b], then there 
is a point p e (a, b) such that 



rb pb 

/ fix)gix)dx = fip) / 



fix)gix)dx = fip) gix)dx. (9.23) 

The proof is omitted. We observe that ix — a)(x — b) does not change sign on 
x e [a, b], so via this theorem we have 



~2Ja ■ 



£>,,,--/ / p '«W)(x-a)(j -i)(fa = -j^/^W-fl) 3 , (9.24) 



where /? e (a, ft). We emphasize that this is the error for n — 1 in Tin). Naturally, 
we want an error expression for n > 1, too. When n > 1, we may refer to the 
integration rule as a compound or composite rule. 

The error committed in numerically integrating over the kth subinterval [xt- i,Xk] 
must be [via (9.24)] 

E k = -4/ (2) (&X** " ^-i) 3 = -T^/ (2) &) = -7^ — V 2) &), (9-25) 
12 12 12 « 

where ^ € [jcji-i, x^] and k = 1, 2, . . . , n — 1, «. Therefore, the total error com- 
mitted is 

£r(„) = / /(*) dx - Tin) = ^ £ t , (9.26) 



which becomes [via (9.25)] 



h b — a -c-^ n , 

E T (n) = ~ — J2 f (&)' ( 9 ' 2? ) 



12 n 

k=\ 
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The average - Yl'l=i /^(£fc) must lie between the largest and smallest values of 
f^ 2 \x) on [a, b], so recalling that f^(x) is continuous on [a, b], the intermediate 
value theorem (Theorem 7.1) yields that there is an f € (a, b) such that 






n 

k=l 

Therefore 

£r ( «) = -^-a)/ (2) (£), (9.28) 

where £ € (a, b). If the maximum value of f^(x) on [a, b] is known, then this 
may be used in (9.28) to provide an upper bound on the error. We remark that 
Erin) is often called truncation error. 
We see that 

T(n) = x T y (9.29) 

for which 

x = h[\\ ■ ■ ■ \\] T € R" +1 , y = [/(* )/(*i) • • • /(*„-i)/(*„)] r e R" +1 - 

(9.30) 
We know that rounding errors will be committed in the computation of (9.29). The 
total rounding error might be denoted by Er. We recall from Chapter 2 [Eq. (2.40)] 
that a bound on these errors is 

\Er\ = \x T y - fl[x T y]\ < 1.01 (n + l)u\x\ T \y\, (9.31) 

where u is the unit roundoff, or else the machine epsilon. The cumulative effect of 
rounding errors can be expected to grow as n increases. If we suppose that 

M= max \f (2) (x)\, (9.32) 

x£[a,b] 



then, from (9.28), we obtain 



1 1 
12^2 



|£r ( „)l<— — (b-afM. (9.33) 



Thus, (9.33) is an upper bound on the truncation error for the composite trapezoidal 
rule. We see that the bound gets smaller as n increases. Thus, as we expect, 
truncation error is reduced as the number of trapezoids used increases. Combining 
(9.33) with (9.31) results in a bound on total error E: 



1 1 

12^2 



\E\ < —^(b - aYM + 1.01 (n + l)w|x|' M- (9.34) 
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Usually n ^> l,son+ 1 ^n. Thus, substituting this and (9.30) into (9.31) results in 

-l 



\x T y - fl[x T y]\ < 1.01(b - a)u 



and so (9.34) becomes 



1 1 , 

\E\ < ,(b - a) 3 M + 1.010 - a)u 

12 n l 



l/(*o)l + !/(*«)! 



£i/(**)i 



*=i 



(9.35) 



|/(x )| + 1/(*„)| 



n-\ 



£l/(**)l 



k=\ 



(9.36) 

In general, the first term in the bound of (9.36) becomes smaller as n increases, 

while the second term becomes larger. Thus, there is a tradeoff involved in choosing 

the number of trapezoids to approximate a given integral, and the best choice ought 

to minimize the total error. 

Example 9.1 We may apply the bound of (9.36) to the following problem. We 
wish to compute 



Jo 



dx. 



Thus, [a, b] — [0, 1], and so b — a = 1. Of course, it is very easy to confirm that 
7=1 — e _1 . But this is what makes it a good example to test our theory out. We 
also see that f (2 \x) = e~ x , and so in (9.32) M = 1, Also 



for£ = 0, 1, 



f(x k ) = e- k ' n 



\,n. It is therefore easy to see that 



n-\ 



J2 f( x k) 



-l/n 



k=\ 



1 



-l/n 



We might assume that the trapezoidal rule for this problem is implemented in the 
C programming language using single-precision floating-point arithmetic, in which 
case a typical value for u would be 

u = 1.1921 x 10" 7 . 

Therefore, from (9.36) the total error is bounded according to 



|£|< 



l 



12n 2 



1.2040 x 10" 



1 



-l/n 



2e 



1 



-l/n 



Figure 9.3 plots this bound versus n, and also shows the magnitude of the computed 
(i.e., the true or actual) total error in the trapezoidal rule approximation, which 
is \T(n) — I\. We see that the true error is always less than the bound, as we 
would expect. However, the bound is rather pessimistic. Also, the bound predicts 
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10 4 10 5 10 6 

Number of trapezoids used (n) 



10 7 




10 u 



10 1 



10 2 10 3 10 4 

Number of trapezoids used (n) 



10 5 



10 b 



Figure 9.3 Comparison of total error (computed) to bound on total error, illustrating the 
tradeoff between rounding error and truncation error in numerical integration by the trape- 
zoidal rule. The bound employed here is that of Eq. (9.36). 



that the proper choice for n is much less than what the computed result predicts. 
Specifically, the bound suggests that we choose n & 100, while the computed result 
suggests that we choose n « 100, 000. 

What is important is that the computed result and the bound both confirm that 
there is a tradeoff between minimizing the truncation error and minimizing the 
rounding error. To minimize rounding error, we prefer a small n, but to minimize 
the truncation error, we prefer a large n. The best solution minimizes the total error 
from both sources. 

In practice, attempting a detailed analysis to determine the true optimum choice 
for n is usually not worth the effort. What is important is to understand the funda- 
mental tradeoffs involved in the choice of n, and from this understanding select a 
reasonable value for n. 



9.3 SIMPSON'S RULE 

The trapezoidal rule employed linear interpolation to approximate f(x) between 
sample points x^ on the x axis. We might consider quadratic interpolation in 
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the hope of improving accuracy still further. Here "accuracy" is a reference to 
truncation error. 

Therefore, we wish to fit a quadratic curve to the points (xk-i, f(xk-\T), 
(xk, f(xk)) an d (xk+i, f{xk+\))- We may define the quadratic to be 



Pk(x) = a(x - Xk) +b(x-Xk) + c. 



(9.37) 



Contrary to past practice, the subscript k now does not denote degree, but rather 
denotes the "centerpoint" of the interval [Xk-i, Xk+i] on which we are fitting the 
quadratic. The situation is illustrated in Fig. 9.4 . For convenience, define v* = 
f(xk)- Therefore, from (9.37) we may set up three equations in the unknowns 
a, b, c: 

a(xk-\ - x k ) 2 +b(xk-i -x k ) + c — y k -\, 

a(xk - x k f + b(x k - Xk ) + C = y k , 

a(x k +i -x k ) 2 +b(xk+i -Xk) + c = yt+i- 

This is a linear system of equations, and we will assume that h — x^ — Xk-i — 
Xk+\ — Xk, so therefore 



b = 



yk+i -2yk + yk-i 
yt+i — yk-i 



2/; 



c = yk- 
This leads to the approximation 



L 



f(x)dx^ / Pk(x) dx = -[y k -i + 4y k + yk+i]. 

Jxt-\ - 1 



(9.38a) 

(9.38b) 
(9.38c) 

(9.39) 



Parabolic arc p k (x) 




x k-1 x k Km 

Figure 9.4 Simpson's rule for numerical integration. 
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Of course, some algebra has been omitted to arrive at the equality in (9.39). As 
in Section 9.2, we wish to integrate fix) on [a, b]. So, as before, a — xq, and 
b — x„. If n is an even number, then the number of subdivisions of [a, b] is an 
even number, and hence we have the approximation 



J a 



f{x) dx 



I pi(x)dx+l pi(x)dx-\ h / p,,- 

JXq J X2 J X n _2 



i (x) dx 



= 3 [yo + 4yi + 2y 2 + 4y 3 +2y 4 + ■ ■ ■ + 2y„_ 2 + 4y„_i + y n ], (9.40) 

The last equality follows from applying (9.39). We define the Simpson rule approx- 
imation to / as 

h 
Sin) = -[y + 4yi + 2y 2 + 4y 3 + 2y 4 + ■ ■ ■ + 2y„- 2 + 4y„-i + y„] (9.41) 



for which n is even and n > 2. 

A truncation error analysis of Simpson's rule is more involved than that of 
the analysis of the trapezoidal rule seen in the previous section. Therefore, we 
only outline the major steps and results. We begin by using only two subintervals 
to approximate I — J f(x) dx, specifically, n — 2. Define c — (a + b)/2. Denote 
the interpolating quadratic by p(x). For a suitable error function e(x), we must 
have 



fix) = p(x) + eix). 



(9.42) 



Immediately we see that 



rb pb pb 

1=1 fix)dx= I pix)dx+ I eix) dx 

Ja Ja Ja 



— — f(a) + 4/ 



/ eix) dx — 5(2) + E S (2)- 

Ja 



fib) 



(9.43) 



So, the truncation error in Simpson's rule is thus 

t-b 



''SO.) 



-I 

J a 



eix)dx. 



(9.44) 



It is clear that Simpson's rule is exact for fix) a quadratic function. Less clear is 
the fact that Simpson's rule is exact if fix) is a cubic polynomial. To demonstrate 
the truth of this claim, we need an error result from Chapter 6. We assume that 
f( k \x) exists and is continuous for all k = 0, 1, 2, 3, 4 for all x € [a, b]. From 
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Eq. (6.14) the error involved in interpolating f(x) with a quadratic polynomial is 
given by 

e(x) = ^/ (3) (f (*))(* " «)(* " b)(x - c) (9.45) 

for some % = %(x) e [a, b]. Hence 

Esm = / «U)d* = ^ / / (3) (£(*))(* - a)(x - b)(x - c)dx. (9.46) 

Unfortunately, polynomial (x — a)(x — b)(x — c) changes sign on the interval 
[a, b], and so we are not able to apply the second mean- value theorem for integrals 
as we did in Section 9.2. This is a major reason why the analysis of Simpson's 
rule is harder than the analysis of the trapezoidal rule. However, at this point we 
may still consider (9.46) for the case where f(x) is a cubic polynomial. In this 
case we must have f^\x) — K (some constant). Consequently, from (9.46) 



iS (2) 



K f h 



a)(x — b){x — c) dx, 



but if z — x — c, then, since c — I (a + b), we must have 



-S(2) 



(b-a) r 



u 

y.!_u b _ a) 



(b-a) I 

(b-a) 



2 
dz. 



dz 



(9.47) 



The integrand is an odd function of z, and the integration limits are symmetric 
about the point z — 0. Immediately we conclude that E$(2) — in this particular 
case. Thus, we conclude that Simpson's rule gives the exact result when f(x) is a 
cubic polynomial. 

Hermite interpolation (considered in a general way in Section 6.4) is polynomial 
interpolation where not only does the interpolating polynomial match f(x) at the 
sample points x^ but the first derivative of f(x) is matched as well. It is useful 
to interpolate f(x) with a cubic polynomial that we will denote by r(x) at the 
points {a, /(«)), (b, f(b)), and (c, /(c)), and also such that r (1) (c) = f {l) (c). A 
cubic polynomial is specified by four coefficients, so these constraints uniquely 
determine r(x). In fact 



for which 



r{x) — p(x) + a(x — a)(x — b)(x — c) 

4[p (1) (c)-/ (1) (c)] 

a — = . 

(b - a) 2 



(9.48a) 



(9.48b) 
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Analogously to (9.19), we may define 

— a)(t — c) 2 (t - b) 

git) = f(t) - r(t) - [f(x) - r(x)]y £ ^7 i- «<'<*• 

(x — a)ix — c) z (x — b) 

(9.49) 

It happens that g^ k \t) for k = 0, 1, 2, 3, 4 all exist and are continuous at all x e 

[a, b]. Additionally, gia) — gib) = gic) — g^ic) — gix) — 0. The vanishing of 

git) at four distinct points on [a, b], and g^'ic) — guarantees that g^ 4 '(£) = 

for some £ € [a, b] by the repeated application of Rolle's theorem. Consequently, 

using (9.49), we obtain 

g (4) (f ) = / ( %) - r (4) (f ) - [/GO - r(*)]- 4! = 0. (9.50) 

(x — a)(x — c) z (x — ») 

Since r(x) is cubic, r^(§) = 0, and so (9.50) can be used to say that 

fix) = rix) + 1/W(|(x))(x - a)(x - c) 2 (x - b) (9.51) 

for x € ia,b). This is valid at the endpoints of [a, b], so finally 

eix) = /(x) - r(x) = i/ (4) (?(x))(x - a)(x - c) 2 (x - b) (9.52) 

for x € [a, b], and f(x) € [a,b]. Immediately, we see that 

E Si 2) = ^f f {A) i^ix))ix-a)ix-c) 2 ix-b)dx. (9.53) 



The polynomial in the integrand of (9.53) does not change sign on [a, b]. Thus, the 
second mean- value theorem for integrals is applicable. Hence, for some £ € ia,b), 
we have 

f (4) (£) f b 
E S (2) = -^^ / ix-a)ix- c) 2 (x -*)</*, (9.54) 

4- J a 

which reduces to 

£5(2) = -^/ (4) (5) (9.55) 

again for some £ € (a, £>), where h — ib — a)/2. 

We need an expression for Es( n ), that is, an error expression for the compos- 
ite Simpson rule. We will assume again that h = ib — a)/n, where n is an even 
number. Consequently 



nb «/ 2 

Es(n) = / /GO d* - Sin) = ^2 / E k 



n/2 

,9.56) 
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where Ek is the error committed in the approximation for the fcth subinterval 
[x 2 {k-\),X2k\- Thus, for & € [x 2 (k-i), xik\, with k=l,2,..., n/2, we have 

Ek = "Lf (4) ^ = -^^^/ (4) &)- (9-57) 

90 90 n 

Therefore 

h 4 h — '^ 2 

E S( „) = V / (4) feO- (9.58) 

w 180 n/2 ^r 

Applying the intermediate-value theorem to the average 4j^j f^iHk) con- 
firms that there is a £ € (a, &) such that 



n/2 * — ' 



jt=i 
so therefore the truncation error expression for the composite Simpson rule becomes 

h 4 
E s(n) = - — (b-a)f^(l;) (9.59) 

for some § € (a,b). For convenience, we repeat the truncation error expression for 
the composite trapezoidal rule: 

£r ( ») = -^-«)/ (2) (?). (9.60) 

It is not really obvious which rule, trapezoidal or Simpson's, is better in general. 
For a particular interval [a,b] and n, the two expressions depend on different 
derivatives of f(x). It is possible that Simpson's rule may not be an improvement 
on the trapezoidal rule in particular cases for this reason. More specifically, a 
function that is not too smooth can be expected to have "big" higher derivatives. 
Simpson's rule has a truncation error dependent on the fourth derivative, while the 
trapezoidal rule has an error that depends only on the second derivative. Thus, a 
nonsmooth function might be better approximated by the trapezoidal rule than by 
Simpson's. In fact, Davis and Rabinowitz [3, p. 26] state that 

The more "refined" a rule of approximate integration is, the more certain we must 
be that it has been applied to a function which is sufficiently smooth. There may be 
little or no advantage in using a 'better' rule for a function that is not smooth. 

We repeat a famous example from Ref. 3 (originally due to Salzer and Levine). 
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Example 9.2 The following series defines a function due to Weierstrass that 
happens to be continuous but is, surprisingly, not differentiable anywhere 2 



00 , 

W(x) = J2~ cos(7 n 7rx). 



(9.61a) 



If we assume that we may integrate this expression term by term, then 

I(y) = f y W(x) dx = -J^-^- sm(Tjry). (9.61b) 

n = \ 

Of course, I = j W(x)dx — lib) — 1(a). The series (9.61b) gives the "exact" 
value for I(y) and so may be compared to estimates produced by the trapezoidal 
and Simpson rules. Assuming that n — 100, the following table of values is obtained 
[MATLAB implementation of (9.61) and the numerical integration rules]: 



Interval 


Exact 


Trapezoidal 


Error 


Simpson's 


Error 


[a,b] 


Value / 


Tin) 


/ - T(n) 


S(n) 


/ - S(n) 


[0,1] 


0.01899291 


0.01898760 


0.00000531 


0.01901426 


-0.00002135 


[.1..2] 


-0.04145650 


-0.04143815 


-0.00001834 


-0.04146554 


0.00000904 


[.2, .3] 


0.03084617 


0.03084429 


0.00000188 


0.03086261 


-0.00001645 


[.3, .4] 


0.00337701 


0.00342534 


-0.00004833 


0.00341899 


-0.00004198 


[.4, .5] 


-0.03298025 


-0.03300674 


0.00002649 


-0.03303611 


0.00005586 



We see that the errors involved in both the trapezoidal and Simpson rules do 
not differ greatly from each other. So Simpson's rule has no advantage here. 



It is commonplace for integration problems to involve integrals that possess 
oscillatory integrands. For example, the Fourier transform of x(t) is defined to be 



X(w) 



/CO fOO POO 

x(t)e~ jcot dt = x(t) cos(o)t) dt - j / x(t) sin(&>f) dt. 

-co J —co J —CO 



(9.62) 
You will likely see much of this integral in other books and associated courses 
(e.g., signals and systems). Also, determination of the Fourier series coefficients 
required computing [recall Eq. (1.45)] 



1 f l7T 
2tt Jo 



(9.63) 



Proof that W(x) is continuous but not differentiable is quite difficult. There are many different 
Weierstrass functions possessing this property of continuity without differentiability. Another example 
complete with a proof appears on pp. 38-41 of Korner [4]. 
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An integrand is said to be rapidly oscillatory if there are numerous (i.e., of the 
order of > 10) local maxima and minima over the range of integration (i.e., here 
assumed to be the finite interval [a, b]). Some care is often required to compute 
these properly with the aid of the integration rules we have considered so far. 
However, we will consider only a simple idea called integration between the zeros. 
Davis and Rabinowitz have given a more detailed consideration of how to handle 
oscillatory integrands [3, pp. 53-68]. 

Relevant to the computation of (9.63) is, for example, the integral 



L 



2?r 

f(x)sm(nx)dx. (9.64) 



It may be that f(x) oscillates very little or not at all on [0, 2tt]. We may therefore 
replace (9.64) with 

2n_1 f(k+l)jr/n 

1=^2 f(x)sm(nx)dx. (9.65) 

k=o ^ k7Z l n 

The endpoints of the "subintegrals" 



J kn In 



•(k+l)Tt/n 

Ik — I f(x)sm(nx)dx (9.66) 

T/fl 



in (9.65) are the zeros of sin(«x) on [0, 2jt]. Thus, we are truly proposing integra- 
tion between the zeros. At this point it is easiest to approximate (9.66) with either 
the trapezoidal or Simpson's rules for all k. Since the endpoints of the integrands 
in (9.66) are zero-valued, we can expect some savings in computation as a result 
because some of the terms in the rules (9.14) and (9.41) will be zero-valued. 



9.4 GAUSSIAN QUADRATURE 

Gaussian quadrature is a numerical integration method that uses a higher order of 
interpolation than do either the trapezoidal or Simpson rules. A detailed derivation 
of the method is rather involved as it relies on Hermite interpolation, orthogonal 
polynomial theory, and some aspects of the theory rely on issues relating to linear 
system solution. Thus, only an outline presentation is given here. However, a 
complete description may be found in Hildebrand [5, pp. 382-400]. Additional 
information is presented in Davis and Rabinowitz [3]. 

It helps to recall ideas from Chapter 6 here. If we know f(x) for x = xu 
where j — 0, 1, . . . , n — 1, n, then the Lagrange interpolating polynomial is (with 
P(xj) = /(*/)) 

n 

p(x) = J^f(xj)Lj(x), (9.67) 

.7=0 
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where 



m*)= n 



X Xj 






Recalling (6.45b), we may similarly define 

n 



i=0 



Consequently 



7r (1) (*) = £ 



n (x - xk) 



k=0 



so that 



Jt m (xi)= Y[(xi-Xk), 



k=Q 



which allows us to rewrite (9.68) as 



7t(x) 

Lj(x) — —-r- 

; jt( v Hxj)(x - xj) 



for j = 0, 1, . . . , n. From (6.14) 

e(x) = fix) - p(x) 



1 



(«+l)! 



/ ( " +1) (|)7r(x) 



(9.68) 



(9.69) 



(9.70) 



(9.71) 



(9.72) 



(9.73) 



for some £ € [a, b], and £ = f (x). 

We now summarize Hermite interpolation (recall Section 6.4 for more detail). 
Suppose that we have knowledge of both f(x) and f^(x) at x = x ; - (again j — 
0, 1, ... , n). We may interpolate f(x) using a polynomial of degree In + 1 since 
we must match the polynomial to both /(x) and /^(x) at x = x,\ Thus, we need 
the polynomial 



pto = ^M*)/(*i0 + £M*)/ (1 W 



(9.74) 



fc=0 



fc=0 



where /ijfc(x) and hk(x) are both polynomials of degree In + 1 that we must deter- 
mine according to the constraints of our interpolation problem. 
If 

hi(xj) = Si-j,hi( X j) = 0, (9.75a) 
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then p(xj) — f(xj), and if we have 

h\ 1 \xj) = 0,hl l \xj) = S i -j, (9.75b) 

then p^\xj) — f^'ixj) for all jf = 0, 1, . . . , n. Using (9.75), it is possible to 
arrive at the conclusion that 



and 



hi(x) = [1 - 2Lf\ Xi )(,x - Xi)][Li(x)f 



h{(x) = (x - Xi)[Li(x)] ■ 



(9.76a) 



(9.76b) 



Equation (9.74) along with (9.76) is Hermite's interpolating formula. [Both parts 
of Eq. (9.76) are derived on pp. 383-384 of Ref. 5, as well as in Theorem 6.1 of 
Chapter 6 (below).] It is further possible to prove that for p(x) in (9.74) we have 
the error function 



1 



(2« + 2)! 



e{x) = f(x) - p(x) = 

where f € [a, b] and f = f (x). 
From (9.77) and (9.74), we obtain 



fix) = J2 hk(x)f(x k ) + J^h k (x)f m (xk) 



f {2n+1) ^)[n{x)f 



(9.77) 



1 



f i2n+2} (Hx))[7t(x)] 2 



k=0 



k=0 



(2n + 2)V 

(9.78) 
Suppose that w(x) > for x e [a, b]. Function w(x) is intended to be a weighting 
function such as seen in Chapter 5. Consequently, from (9.78) 



" r rb 



j w(x)f(x)dx=') I w(x)h k (x)dx 

Ja k=Q 



= ^ / w{x)h k (x)dx f(x k ) 

,._ Ua J 

^2 / w{x)h k (x)dx f (1) (x k ) 

_L^ /" fV»+z>(£( x )) w (x)[„(x)?dx, (9.79) 

« + 2)! J fl 



*=o 



(2k 



where a < f (x) < b, if a < x k < b. This can be rewritten as 

rb " n 

I w(x)f(x) dx = Yl H k f(x k ) + J2 H k f m (x k ) + E, 



k=0 



k=0 



(9.80) 
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where 

f b 

H k — I w(x)h k (x)dx 

J a 

and 



= J w(x 

J a 



)[l-2L^\x k Kx-Xk)][L k (x)] 2 dx (9.81a) 



H k 



- 

J a 



w(x)h k (x) dx 



r 

= I w(x 

J a 



){x - x k )[L k (x)] dx. 



(9.81b) 



If we neglect the term E in (9.80), then the resulting approximation to f w(x) 
f(x)dx is called the Hermite quadrature formula. Since we are assuming that 
w(x) > 0, the second mean-value theorem for integrals allows us to claim that 



E = 



1 



(2« + 2)! 



i 



f (2n+2) (H) I w{x)[n(x)Y dx 



(9.82) 



for some f € [a, b]. 

Now, recalling (9.72), we see that (9.81b) can be rewritten as 



w(x)(x — x k ) 

b 



H k = [ 

J a 

1 f 

= m - , / W(X)7T 
X W (x k ) Ja 



7t(x) 



_7T {l >(x k )(x -X k ). 



dx 



(A') 



7T(x) 



_n (1) (x k )(x -x k ) 

1 f b 

— — 7TT- ~ w(x)7T(x)L k (x)dx. 

Tt W (x k ) J a 



dx 



(9.83) 



We recall from Chapter 5 that an inner product on L [a, b] is (/, g are real-valued) 

-b 



(/.«) 



/ 

J a 



w(x)f(x)g(x)dx. 



(9.84) 



Thus, H k = for k = 0, 1, . . . , n if n(x) is orthogonal to L k (x) over [a, b] with 
respect to the weighting function w(x). Since deg(L^(x)) = n (all k), this will be 
the case if it(x) is orthogonal to all polynomials of degree < n over [a, b] with 
respect to the weighting function w(x). Note that deg(7r(x)) = n + 1 [recall (9.69)]. 
In fact, the polynomial jt(x) of degree n + 1 is orthogonal to all polynomials 
of degree < n over [a,b] with respect to w(x), the Hermite quadrature formula 
reduces to the simpler form 



nb » 

/ w(x)f(x)dx = ^ / H k f(x k ) + E, 

Ja fc=o 



where 



E = 



1 



(2« + 2)! 



f b 

f i2n+2 \H) / W(x)[7t(x)] 2 dx, 
Ja 



(9.85) 



(9.86) 
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and where xq, x\, . . . , x„ are the zeros of n(x) (such that a < x k < b). A formula of 
this type is called a Gaussian quadrature formula. The weights H k are sometimes 
called Christoffel numbers. We see that this numerical integration methodology 
requires us to possess samples of f(x) at the zeros of it(x). Variations on this 
theory can be used to remove this restriction [6], but we do not consider this 
matter in this book. However, if f(x) is known at all x e [a, b], then this is not a 
serious restriction. 

In any case, to apply the approximation 

/ w(x)f(x)dx^J^H k f(x k ), (9.87) 

Ja k=0 

it is clear that we need a method to determine the Christoffel numbers H k . It is 
possible to do this using the Christoffel-Darboux formula [Eq. (5.11); see also 
Theorem 5.2]. From this it can be shown that 

Hk = fn^n+2 ^ (9g8) 

<t>n+l,n+l^ n+ i(Xk)<t>n+l(Xk) 

where polynomial (j> r {x) is obtained, for instance, from (5.5) and where 

<t>n+i(x) = </>„ + i,„+i7r(x). (9.89) 

Thus, we identify the zeros of it(x) with the zeros of orthogonal polynomial 
<p n+ i(x) [recalling that {</>,-, <pj) — &i-j with respect to inner product (9.84)]. 

Since there are an infinite number of choices for orthogonal polynomials <j>k(x) 
and there is a theory for creating them (Chapter 5), it is possible to choose n(x) in 
an infinite number of ways. We have implicitly assumed that [a, b] is a finite length 
interval, but this assumption is actually entirely unnecessary. Infinite or semiinfinite 
intervals of integration are permitted. Thus, for example, tc(x) may be associated 
with the Hermite polynomials of Section 5.4, as well as with the Chebyshev or 
Legendre polynomials. 

Let us consider as an example the case of Chebyshev polynomials of the first 
kind (first seen in Section 5.3). In this case 

1 

w(x) — 



VT^x 1 

with [a, b] — [—1, 1], and we will obtain the Chebyshev -Gauss quadrature rule. 
The Chebyshev polynomials of the first kind are T k {x) — cos[£cos _1 x], so, via 
(5.55) for k > 0, we have 

<Pk(x) = J-T k (x). (9.90) 

V n 
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From the recursion for Chebyshev polynomials of the first kind [Eq. (5.57)], we 
have (k > 0) 

(9.91) 



n,k 



,*-! 



(T k (x) = j: k j=0 T k jxj).Thus 



We have that T k (x) = for 



<Pk,k — \ — 2 
V it 



k-\ 



(9.92) 



X — Xj — cos 



2i + 1 
2k 



(9.93) 



(i = 0, 1, . . . , k - 1). Additionally 



77 (;c) = k sin[k cos x] 



vT^^2 



(9.94) 



so, if, for convenience, we define a,- = =^-7t, then xt — cos a,-, and therefore 



T, {l \ Xi ) = k 



sin (to,) 
sin a, 



2i + l 



sm a,- 



sin «,■ 



(9.95) 



Also 

Z*+i(*i) = cos 



, 2j + 1 

(* + 1) I ~^— ?T 



2/< 



= cos 



2/ + 1 2/ + 1 



2* 



2i + 1 

7T 

2 

2i + 1 



2i + 1 
2£ 



2i + 1 



2j + 1 



= — sin 



sin a, = (—1) sin a,-. 



i+i 



(9.96) 



Therefore, (9.88) becomes 



FT 

l± 2 n+l 



H k = 



TT 



/2 \ / \2 

'-2» [ /_(n+l) 

7T / \ V 7T 



sina^ 



'2 \ «+ 1 

_(_!)*+ 1 s i nai( , 

TT 



(9.97) 



Thus, the weights (Christoffel numbers) are all the same in this particular case. So, 
(9.87) is now 

\ -4==dx^ V f cos it) =C(n). (9.98) 



TLFeBOOK 



GAUSSIAN QUADRATURE 



391 



The error expression E in (9.86) can also be reduced accordingly, but we will omit 
this here (again, see Hildebrand [5]). 

A simple example of the application of (9.98) is as follows. 



Example 9.3 Suppose that f(x) = Vl — x 2 , in which case 



/ 



For this case (9.98) becomes 
■ l 



: dx 



— J dx — 



2. 



/I 
dx « C(n) — V^ sin 



2k + 1 
2n + 2' 



For various n, we have the following table of values: 



n 


C(n) 


1 


2.2214 


2 


2.0944 


5 


2.0230 


10 


2.0068 


20 


2.0019 


100 


2.0001 



Finally, we remark that the error expression in (9.82) suggests that the method 
of this section is worth applying only if f(x) is sufficiently smooth. This is con- 
sistent with comments made in Section 9.3 regarding how to choose between the 
trapezoidal and Simpson's rules. In the next example f(x) — e~ x , which is a very 
smooth function. 



Example 9.4 Here we will consider 



1 * 1 

e~ x dx = e - - = 2.350402387 

1 e 



and compare the approximation to / obtained by applying Simpson's rule and 
Legendre- Gauss quadrature. We will assume n — 2 in both cases. 

Let us first consider application of Simpson's rule. Since xq — a — —\,x\ = 
0,xi=b=l(h = (b- a) In = (1 - (-l))/2 = 1), we have via (9.41) 

5(2) = -[e +l + 4e° + e' 1 ] = 2.362053757 
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for which the error is 

E S (2) = / - S(2) = -0.011651. 

Now let us consider the Legendre- Gauss quadrature for our problem. We recall 
that for Legendre polynomials the weight function is w(x) — 1 for all x e [—1, 1] 
(Section 5.5). From Section 5.6 we have 

i /7 1 , 

fo(x) = P 3 (x) = J — [5x 3 - 3x], 

\\Pi\\ V 22 



1 /9 1 A , 

— P 4 (x) = ,/ — [35jc 4 - 30x 2 + 3]. 
3 4 || V 28 



d>4(x) - 

\\P 4 \\ 

Consequently, $33 = jJj, and the zeros of cpsix) are at* = 0, =W§, so now our 
sample points (grid points, mesh points) are 

[J [3 

xo = -J -, xi = 0, x 2 — +J -. 



Hence, since 0^ (x) = ./^[lS* 2 — 3], we have 



i>?\x ) = 3^, ^ X) (xi) = -^, 4 1 \x 2 ) = 3^. 



Also, 04,4 = g ■%/§. an d 



3/9 3/9 3/9 

4>4( x o) — \ —, 4>4( x l) — —\ —, 4>4(. x 2) — \ —• 

w 10V 2 8V2 10V 2 

Therefore, from (9.88) the Christoffel numbers are 

5 8 5 

Ho =-,Hi = -, H 2 = -. 



From (9.87) the resulting quadrature is 



£ 



with 



e~ x dx « //o<?"*° + ^l^"* 1 + #2e~* 2 = L(2) 



L(2) = 2.350336929, 
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and the corresponding error is 

E L( 2) = I - L(2) = 6.5458 x 10" 5 . 

Clearly, |£"l(2)| <SC \Es(2)\- Thus, the Legendre-Gauss quadrature is much more 
accurate than Simpson's rule. Considering how small n is here, the accuracy of the 
Legendre-Gauss quadrature is remarkably high. 

9.5 ROMBERG INTEGRATION 

Romberg integration is a recursive procedure that seeks to improve on the trape- 
zoidal and Simpson rules. But before we consider this numerical integration 
methodology, we will look at some more basic ideas. 

Suppose that I = f f(x) dx and I(n) is a quadrature that approximates /. For 
us, I(n) will be either the trapezoidal rule T(n) from (9.14), or else it will be 
Simpson's rule S(n) from (9.41). It could also be the corrected trapezoidal rule 
Tc(n), which is considered below [see either (9.104), or (9.107)]. 

It is possible to improve on the "basic" trapezoidal rule from Section 9.2. Begin 
by recalling (9.27) 

1 " 

E T(n) = --h 3 J2f (2) ^k), (9.99) 

k=\ 

where h — (b — a) In, xq — a, x n — b, and ^ € [xk—i, Xfc]. Of course, Xk — xt-i — 
h (uniform sampling grid). Assuming (as usual) that f w (x) is Riemann integrable 
for all k > 0, then 

lim y>/ (2) (&) = / (1) O>)-/ (1) (0)= / f (2) (x)dx. (9.100) 

Thus, we have the approximation 

n 

J2 h f (2) ^k) * f W (b) - f m (a). (9.101) 

k=l 

Consequently, 

1 " 1 

E T (n) = -—h 2 J2 h f (2) ^k) » -— h 2 [f m (b) - f m (a)] (9.102) 

k=\ 



I _ T (n) w -±h 2 [f m (b) - f W (a)l (9.103) 
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This immediately suggests that we can improve on the trapezoidal rule by replacing 
T(n) with the new approximation 

T c (n) = T(n) - ±^h 2 [f m (b) - f m (a)], (9.104) 

where Tc(n) denotes the corrected trapezoidal rule approximation to /. Clearly, 
once we have T(n), rather little extra effort is needed to obtain Tc(n). 

In fact, we do not necessarily need to know f^(x) exactly anywhere, much 
less at the points x — a or x — b. In the next section we will argue that either 

f W (x) = i-3/Oc) + 4f(x + h)- f{x + 2/z)] + V/ (3) (?) (9.105a) 
2h 3 

or that 

f W (x) = ^-[3 fix) - 4f(x -h) + f{x - 2/0] + \h 2 f (i \H). (9.105b) 
2/z 3 

In (9.105a) f e [x, x + 2h], while in (9.105b) £ € [x - 2h, x]. Consequently, with 
xq — a for ^o € [a, a + 2h], we have [via (9.105a)] 

fW(xd) = f m (a) = -L[-3/(x ) +4/(xj) - f(x 2 )] + V/ (3) (?o), (9.106a) 
2/z 3 

and with x„ = b for £„ € [b — 2h, b], we have [via (9.105b)] 

f W (Xn) = / (1) W = ^r[3/fc) - 4/(x„-l) + f(x n - 2 )] + \h 2 f^^ n ). 

2h 3 

(9.106b) 

Thus, (9.104) becomes {approximate corrected trapezoidal rule) 

T c (n) = T(n) - ^[3/(x„) - 4/(x„_i) + /(*„_ 2 ) + 3/(x ) - 4/(x x ) + /(x 2 )] 

/i 4 
-Tr[/ (3) fe)-/ (3) (fo)]. (9.107) 

36 

Of course, in evaluating (9.107) we would exclude the terms involving f^H^o), 
and/< 3 >(£„). 

As noted in Epperson [7] for the trapezoidal and Simpson rules 

1 
I-I(n)cx— , (9.108) 

nP 

where p = 2 for /(n) = T(n) and p = 4 for /(n) = ,!>(«). In other words, for a 
given rule and a suitable constant C we must have I — I(n) ^ Cn~ p . Now observe 
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that we may define the ratio 



l'4n 



I(n) - I (In) 



(I - Cn~P) - (I - C(2n)~P) 



I(2n) 
(2n)~ 



- I (An) 
P -n~P 



(I 



(4n)~ 



(2nY 



C(2n)~P) 
2-P - 1 
A~p - 2-P 



(I - C(4n)~P) 

= 2 p . 



Immediately we conclude that 



log 2 r 4n = 



lQ g W r 4n 

log 10 2 ' 



(9.109) 



(9.110) 



This is useful as a check on program implementation of our quadratures. If (9.110) 
is not approximately satisfied when we apply the trapezoidal or Simpson rules, 
then (1) the integrand f(x) is not smooth enough for our theories to apply, (2) 
there is a "bug" in the program, or (3) the error may not be decreasing quickly 
with n because it is already tiny to begin with, as might happen when integrating 
an oscillatory function using Simpson's rule. 

The following examples illustrate the previous principles. 

Example 9.5 In this example we consider approximating 



-i 



jr/2 



sin x dx = 1 



using T(n) [via (9.14)], T c (n) [via (9.104)], and S(n) [via (9.41)]. Parameter p in 
(9.1 10) is computed for each of these cases, and is displayed in the following table 
(where "NaN" means "not a number"): 



T(n) 



p for T(n) 



T C (n) 



p for T c (n) 



S(n) 



p for S(n) 



2 


0.94805945 


NaN 


0.99946364 


NaN 


1.00227988 


NaN 


4 


0.98711580 


NaN 


0.99996685 


NaN 


1.00013458 


NaN 


8 


0.99678517 


2.0141 


0.99999793 


4.0169 


1.00000830 


4.0864 


16 


0.99919668 


2.0035 


0.99999987 


4.0042 


1.00000052 


4.0210 


32 


0.99979919 


2.0009 


0.99999999 


4.0010 


1.00000003 


4.0052 


64 


0.99994980 


2.0002 


1.00000000 


4.0003 


1.00000000 


4.0013 


128 


0.99998745 


2.0001 


1.00000000 


4.0001 


1.00000000 


4.0003 


256 


0.99999686 


2.0000 


1.00000000 


4.0000 


1.00000000 


4.0001 



We see that since f(x) — sinx is a rather smooth function we obtain the values 
for p that we expect to see. 

Example 9.6 This example is in contrast with the previous one. Here we 
approximate 



Jo 



/o 4 



TLFeBOOK 



396 



NUMERICAL INTEGRATION AND DIFFERENTIATION 



using T(n) [via (9.14)], T c (n) [via (9.107)], and S(n) [via (9.41)]. Again, p 
[via (9.110)] is computed for each of these cases, and the results are tabulated 
as follows: 



T(n) 



p for T{n) 



T C (n) 



p for T c (n) 



S(n) 



p for S(n) 



2 0.64685026 


NaN 


0.69580035 


NaN 


0.69580035 


NaN 


4 0.70805534 


NaN 


0.72437494 


NaN 


0.72845703 


NaN 


8 0.73309996 


1.2892 


0.73980487 


0.8890 


0.74144817 


1.3298 


16 0.74322952 


1.3059 


0.74595297 


1.3275 


0.74660604 


1.3327 


32 0.74729720 


1.3163 


0.74839388 


1.3327 


0.74865310 


1.3332 


64 0.74892341 


1.3227 


0.74936261 


1.3333 


0.74946548 


1.3333 


128 0.74957176 


1.3267 


0.74974705 


1.3333 


0.74978788 


1.3333 


256 0.74982980 


1.3292 


0.74989962 


1.3333 


0.74991582 


1.3333 


We observe that 


/(*) = 


x 1 / 3 , but that f m 


(x) = ttX" 


-2/3 >/ (2) W = 


_2 x -5/3 



etc. Thus, the derivatives of f(x) are unbounded at x = 0, and so f(x) is not 
smooth on the interval of integration. This explains why we obtain p « 1.3333 in 
all cases. 

As a further step toward Romberg integration, consider the following. Since 
/ - I(2n) « C(2n)~ p = C2~ p n~ p « 2~ P (I - /(«)), we obtain the approximate 
equality 

_ I(2n)-2- p I(n) 
1 -2-p ' 



or 



2 p I (2n) - I (n) 
I % K —: K —L = R(2n). 

2P - i 



(9.111) 



We call R(2n) Richardson's extrapolated value (or Richardson's extrapolation), 
which is an improvement on I(2n). The estimated error in the extrapolation is 
given by 

I(n) - I(2n) 
E R(2n) = I{2n)-R{2n)= 2 „ _\ ■ (9.H2) 



Of course, p in (9.111) and (9.112) must be the proper choice for the quadrature 
I(n). We may "confirm" that (9.111) works for T(n) (as an example) by considering 
(9.28) 

h -(b-a)fV>(S) (9.113) 



<T(n) 



12 



(for some f € [a, b]). Clearly 



(hl2Y m 1 
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7-r(2»)wi[/-r(n)], 



47(2w) - 7(«) 
/ « — — — = /?t(2w), 



(9.114) 



which is (9.111) for case p — 2. Equation (9.114) is called the Romberg integration 
formula for the trapezoidal rule. Of course, a similar expression may be obtained 
for Simpson's rule [with p = 4 in (9.111)]; that is, for n even 



_ 165(2») - 5(«) ^ 

/ s» = Rs(2n). 

15 A 



In fact, it can also be shown that Rj(2n) — S(2n): 

4T(2n) - T(n) 



5(2/i) = 



3 



(9.115) 



(9.116) 



(Perhaps this is most easily seen in the special case where n — 1.) In other words, 
the Romberg procedure applied to the trapezoidal rule yields the Simpson rule. 

Now, Romberg integration is really the repeated application of the Richardson 
extrapolation idea to the composite trapezoidal rule. A simple way to visualize the 
process is with the Romberg table {Romberg array): 



7(1) 










7(2) 


S(2) 








7X4) 


S(4) 


Rs(4) 






7(8) 


5(8) 


Rs(8) 


9 




7(16) 


5(16) 


R S (16) 


9 


9 



In its present form the table consists of only three columns, but the recursive 
process may be continued to produce a complete "triangular array." 

The complete Romberg integration procedure is often fully justified and devel- 
oped with respect to the following theorem. 



Theorem 9.1: Euler-Maclaurin Formula Let / € C + [a, b] for some 
k > 0, and let us approximate I — f f(x)dx by the composite trapezoidal rule 
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of (9.14). Letting h n — (b — a)/n for n > 1, we have 

T ^ = 1 + E ^f[f^~ l Hb) - / (2 '-%)] 

^ (20! 

fi 2*+ 2 r2k+2 rh s f (2k+2) 



(2k + 2)\ 
where r\ e [a, b], and for j > 1 

2 



h^(b-a)f (JX ^'(n), (9.117) 



% = (-1) 



/-i 



E 



(2jtm) 2 ' 

.n=l 



(2;)! (9.118) 



are the Bernoulli numbers. 



Proof This is Property 9.3 in Quarteroni et al. [8]. A proof appears in Ralston 
[9]. Alternative descriptions of the Bernoulli numbers appear in Gradshteyn and 
Ryzhik [10]. Although not apparent from (9.118), the Bernoulli numbers are all 
rational numbers. 

We will present the complete Romberg integration process in a more straight- 
forward manner. Begin by considering the following theorem. 

Theorem 9.2: Recursive Trapezoidal Rule Suppose that h = (b — o)/(2n); 
then, for n > 1 (xq — a , x n — b) 

1 " 

T(2n) = -T(n) + h^ /(*0 + (2* - V)h). (9.119) 

k=\ 

The first column in the Romberg table is given by 

TV) = \ T (2^) 4- *g« £ / (xo 4- (2fe - 1) 2 ( ;"- X0) ) (9.120) 

for all n > 1. 

Proof Omitted, but clearly (9.120) immediately follows from (9.119). 

We let R„ denote the row n, column k entry of the Romberg table, where k — 
0, 1, . . . , N and n — k, . . . , N [i.e., we construct an (N + I) x (N + 1) lower trian- 
gular array, as suggested earlier]. Table entries are "blank" for n = 0, 1, . . . , k — 1 
in column k. The first column of the table is certainly 

R (P) = T( 2") (9.121) 
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forn = 0, 1, 



N. From Theorem 9.2 we must have 



Ri Q) = ±R?\ 



r,(0) _ 



2»-i 
On Z—r ^ I 

*=1 v 



(2fc- l)(fc -a) 



2" 



(9.122a) 



(9.122b) 



for m = 1, 2, . . . , Af. Equations (9.122) are the algorithm for constructing the first 
column of the Romberg table. The second column is R„ , and these numbers are 
given by [via (9.114)] 



R m = 



4R 



«)) 



R 



(0) 
n-\ 



1 



(9.123) 



,(2) 



for n = 1, 2, . . . , N. Similarly, the third column is R„ , and these numbers are 
given by [via (9.115)] 



R 



(2) 



4 2 Ri l) 



R 



(i) 



(9.124) 



for n = 2,3, . 
according to 



4 2 -l 
N. The pattern suggested by (9.123) and (9.124) generalizes 



R (k) = 



4*tff- 1) 



7? 



M-l 



4* - 1 



(9.125) 



for n = fc, . . . , N, with k = 1, 2, . . . , JV, Assuming that /(*) is sufficiently smooth, 
we can estimate the error using the Richardson extrapolation method in this manner: 



E^> = 



(k) 



(k) _ «-l 



R 



(k) 



4 k 



(9.126) 



(/<) 



This can be used to stop the recursive process of table construction when E„ 

(k) 

is small enough. Recall that every entry R n in the table is an estimate of I = 
f f(x)dx. In some sense R N is the "final estimate," and will be the best one if 
f(x) is smooth enough. Finally, the general appearance of the Romberg table is 



r,(0) 

^0 










< 


< 








<> 


4 1} 


jf> 






p(0) 


r,(l) 

K N-1 


r,(2) 

K N-\ 


r.(A'-l) 

K N-1 




n(0) 
K N 


r,(D 

K N 


r,(2) 


R N 


K N 
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The Romberg integration procedure is efficient. Function evaluation is confined 
to the construction of the first column. The remaining columns are filled in with 
just a fixed (and small) number of arithmetic operations per entry as determined 
by (9.125). 

Example 9.7 Begin by considering the Romberg approximation to 



/ 

Jo 



e x dx = 1.718281828. 



The Romberg table for this is (N — 3) 



1.85914091 
1.75393109 
1.72722190 
1.72051859 



1.71886115 
1.71831884 
1.71828415 



1.71828269 
1.71828184 



1.71828183 



(3) 

Table entry R\ — 1.71828183 is certainly the most accurate estimate of /. Now 
contrast this example with the next one. 

The zeroth-order modified Bessel function of the first kind Ioiy) € R(y G R) 
is important in applied probability. For example, it appears in the problem of 
computing bit error probabilities in amplitude shift keying (ASK) digital data com- 
munications [11]. There is a series expansion expression for Iq(j), but there is also 
an integral form, which is 



h(y) 



2tt Jo 



2;t 



,)>cosa: 



dx. 



(9.127) 



For y = 1, we have (according to MATLAB's besseli function; Io(y) =besseli(0,y)) 

/o(l) = 1.2660658778. 
The Romberg table of estimates for this integral is (N — 4) 



2.71828183 










1.54308063 


1.15134690 








1.27154032 


1.18102688 


1.18300554 






1.26606608 


1.26424133 


1.26978896 


1.27116647 




1.26606588 


1.26606581 


1.26618744 


1.26613028 


1.26611053 



Plainly, table entry R 



(4) 



1.26611053 is not as accurate as R 



(0) 



1.26606588. 



Apparently, the integrand of (9.127) is simply not smooth enough to benefit from 
the Romberg approach. 
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9.6 NUMERICAL DIFFERENTIATION 



A simple theory of numerical approximation to the derivative can be obtained via 
Taylor series expansions (Chapter 3). Recall that [via (3.71)] 



f(x + h) = J2 TT' (t) ( *> + j! —7rJ in+V} ^ 



k=Q 



k\ 



(n+1)!' 



(9.128) 



for suitable f € [x, x + h]. As usual, 0! = 1, and fix) — f^(x). Since from ele- 
mentary calculus 



f a) (x) = lim 



fix + h)- f(x) (1) f( x + h )-f(x) 



f w (x) 



from (9.128), we have 

/ (1; Cv) - 



/is fix + h) — fix) 1 n ~. 
( 1 )/- T .^ _ Li I >_ ■> v ' hf"'(i;). 



2! 



This was obtained simply be rearranging 



fix + h) = fix) + hf {l \x) + Ift 2 / (2) (?)- 



We may write 



/ (1) (x) 



fix + h)- fix) 



2! 



hf (2 \H) 



-ff\x) 



-et(x) 



(9.129) 



(9.130) 



(9.131) 



(9.132) 



?(i), 



Approximation f\ (x) is called the forward difference approximation to f ( '(x), 
and the error efix) is seen to be approximately proportional to h. Now consider 
(fi € [x,x + h]) 

fix + h) = fix) + hf {l \x) + ^h 2 f (2 \x) + Ift 3 / (3) (Si), (9.133) 

and clearly (£2 e [x — h, x]) 

fix -h) = fix) - hf (l \x) + ^h 2 f (2 \x) - Ift 3 / (3) fe). (9.134) 



(9.134) 


f m ix) - 


fix) - fix - 

h 


-H) + 


^/ (2) (S) 




=/T« 




=e h (x) 



(9.135) 
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?(1)/ 



Here the approximation /, (x) is called the backward difference approximation to 
f^ix), and has error eb(x) that is also approximately proportional to ft. However, 
an improvement is possible, and this is obtained by subtracting (9.134) from (9.133) 



r(D 



1 



f(x + ft) - f(x - ft) = 2hf w ix) + -W(§i) + / w (| 2 )] 



3rfO)i 



c(3). 



3! 



or on rearranging this, we have 



r(D 



f w ix) = 



fix + h)-fix- 
2ft 


-« + 


^/ (3) (fl) + / (3) fe) _ 

6 2 




** 




./."'w 


=«*(*) 



(9.136) 



Recalling the derivation of (9.28), there is a £ € [x — ft, x + ft] such that 



/ (3) (?)=2 [/(3)(?l) + /(3)fe)] 



(9.137) 



(fi € [jc, jc + ft], §2 € [x — ft, ft]). Hence, (9.136) can be rewritten as (for some 
£ € [x — ft, x + ft]) 



/ (1) (x) 



/(* + ft) - /(* - 


-A) 


^ 2 m 
6 


2ft 




=/ c (1) M 




=e c W 



(9.138) 



Clearly, the error e c ix), which is the error of the central difference approximation 
fc \x) to f^ix), is proportional to ft 2 . Thus, if fix) is smooth enough, the 
central difference approximation is more accurate than the forward or backward 
difference approximations. 

The errors e/ix), eb(x), and e c ix) are truncation errors in the approximations. 
Of course, when implementing any of these approximations on a computer there 
will be rounding errors, too. Each approximation can be expressed in the form 



?(!)/ 



1 



r>(x)= -f'y, 

ft 



where for the forward difference approximation 

/ = [/(*) f(x + h)f,y = [-l if, 

for the backward difference approximation 

/ = [/(* -ft) fix)] T ,y = [-l if, 



(9.139) 



(9.140a) 



(9.140b) 
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and for the central difference approximation 

/ = [/(* -A) f{x+h)f,y=\[-\ if. (9.140c) 

Thus, the approximations are all Euclidean inner products of samples of f(x) with 
a vector of constants y, followed by division by h. This is structurally much the 
same kind of computation as numerical integration. In fact, an upper bound on the 
size of the error due to rounding is given by the following theorem. 

Theorem 9.3: Since fl[f m (x)] = // [ //[ { Ty] l, we have (/, y e R m ) 

\fl[f m (x)] - f m (x)\ < j[\Mm + \]\\f\\ 2 \\y\\2- (9.141) 

h 

Proof Our analysis here is rather similar to Example 2.4 in Chapter 2. Thus, 
we exploit yet again the results from Chapter 2 on rounding errors in dot product 
computation. 



Via (2.41) 



for which 



In addition 



//[/ r y] = / r y(i + 6i) 



|ei| < l.Olmu- 



\f T y\ 



//[/(D (x)] = ^-Z(l + ei )(l + e), 

h 

where |e| < u. Thus, since 

~m f T y f T y 

fl[f a \x)] = LJL + LJL {€l + e + eie)) 

h h 

we have 

\flU {l \x)} - f m (x)\ < \L2i(\ €l \ + | e | + | ei || e |) = \LA 

h h 

\f\ T M , , ,„,.... i\f\ T \y\ 



x ( l.Olmu- — = \-u + l.Olmu 



\f T y\ ' \f T y\ 

and since u 2 <$C u, we may neglect the last term, yielding 

\fl[f W (x)] - f m (x)\ < X - (l.01m M |/| r |y| + u\fy\) 
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But |/fM=<|/|,|yl><ll/ll2l|y||2, and \fy\ = |</, y)\ < ||/|| 2 ||y||2 via 
Theorem 1.1, so finally we have 

\fl[f m ix)] - / (1) (*)l < jH.Olm + l]||/||2||y||2, 

h 

which is the theorem statement. 

The bound in (9.141) suggests that, since we treat u, m, ||/||2, and | |y| I2 are 
fixed, as h becomes smaller, the rounding errors will grow in size. On the other 
hand, as h becomes smaller, the truncation errors diminish in size. Thus, much as 
with numerical integration (recall Fig. 9.3), there is a tradeoff between rounding 
and truncation errors leading in the present case to the existence of some optimal 
value for the choice of h. In most practical circumstances the truncation errors will 
dominate, however. 

We recall that interpolation theory from Chapter 6 was useful in developing 
theories on numerical integration in earlier sections of the present chapter. We 
therefore reasonably expect that interpolation ideas from Chapter 6 ought to be 
helpful in developing approximations to the derivative. 

Recall that p n (x) — Yl'l=o Pn,kX k interpolates fix) forx € [a, b], i.e., p n ix k ) = 
f(xk) with xq — a, x n — b, and x k e [a, b] for all k, but such that Xk ^ x; for 
k # j. If f(x) e C n+l [a, b], then, from (6.14) 

1 " 

fix) = p„ix) + ——T-J (n+l) &) \\(x - Xi) (9.142) 

(„ + l)! fj 

(| = %ix) € [a, b]). For convenience we also defined nix) — Y\'!=o^ x ~ x i)- Thus, 
from (9.142) 



f m ix) = pi 1 \x) + - 



ff (i) W /(»+i)(|) + wW A / (»+i) ( |) 



(«+!)![ ' dx' 



(9.143) 



Since % = % ix) is a function of x that is seldom known, it is not at all clear 
what df( n+ ^i%)/dx is in general, and to evaluate this also assumes that d^ix)/dx 
exists. 3 We may sidestep this problem by evaluating (9.143) only for x = Xk, and 
since nixk) — for all k, Eq. (9.143) reduces to 

f (i) (x k ) = pi'Hxk) + - l —-, n m ixk)f (n+l) i^Xk)). (9.144) 

in + 1)! 

For simplicity we will now assume that Xk — xq + hk, h — ib — a)/n (i.e., 
uniform sampling grid). We will also suppose that n — 2, in which case (9.144) 
becomes 

f m ix k ) = p^ixk) + l -Jt {l \x k )f & i^ix k )). (9.145) 

This turns out to be true, although it is not easy to prove. Fortunately, we do not need this result. 
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Since tt(x) — (x — xo)(x — x\){x — X2), we have jr^Cr) = (x — x\)(x — x 2 ) + 
(x — xq)(x — xi) + (x — xq)(x — X2), and therefore 



(9.146) 





7r m (x ) = 


C*0 


-Xl)(XQ 


-X2) = 


(-A)(-2A) 


= 2/j 2 , 




7T (1) (*l) = 


(x\ 


-xo)(x\ 


-X2) 


= h(-h) 


= -^ 2 , 




7T (1) (X 2 ) = 


(X2 


-x )(x 2 


-xi) 


= (2A)/i 


= 2/z 2 . 


From 


(6.9), we obtain 












P2(x) = 


f(xo)Lo< 


(x) + f(x 


i)Li(x) + /(x 2 )L 2 (x), 


where 


, from (6.11) 




Lj( 


»=n 







(=0 "■' *' 



^2W - (jt 2 -jc )(jc 2 -^l) ' ^2 W - (jt 2 -*0)(*2-*l)- 



Hence 



(1)/-. \ _ J r(l)/-. \ _ 1 ,(1), 



( l \x ) = ,lP(xi) = + — ,4 1) (x 2 )= T — . 

2 v u; 2/i ' 2 2/j 2 <■ ^ ^ 2 /j 



(9.147) 



(9.148) 



and hence the approximation to f^(xk) is (k e {0, 1, 2}) 

P ( n\x k ) = f(x )L$\x k ) + /(jcOLJ^xt) + /(x 2 )L^ 1) (x^). (9.149) 

From (9.148) 

j „/„\ _ (*-*l)(*-*2) r (1)/- t n _ (*-*l)+(*~*2) 

^0W - (* -*l)(*0-*2)' W - Cc -*i)(*0-*2)' 

L t (x)= ('-*>)(*-*2) L< 1} (x)= (x-xo)+(x-X2) (9150) 

1V ' (*l-*o)(*l-*2)' 1 V 7 (JCI— JC0)(^1— ^2)' V ' 

_ (x-x )(x-x l ) j (1)/ \ _ (x-*o)+Q:-:ti) 



Li, l; (x ) = — -, U^(xi) = — -, L' i '(x 2 ) = + — , 
" 2A ° 2h ° 2h 

< l i\x6) = +7, L^Cxi) = 0, L ( 1 1) (x 2 ) = -p (9.151) 

3 



We therefore have the following expressions for f^(xk)'. 

f (i) (x ) = i-[-3/(* ) + 4/(xi) - /(x 2 )] + V/( 3 >(f (x )), (9.152a) 

2« 3 

/ (1) (*i) = ^-[-/(Jto) + f(xi)] - -U 2 / (3) (£(*i)), (9.152b) 

2« 6 

/ (1) fe) = 4;[/(*o) - 4/(xi) + 3/(x 2 )] + V/ (3) (?fe)). (9.152c) 

2« 3 
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We recognize that the case (9.152b) contains the central difference approximation 
[recall (9.138)], since we may let xq — x — h, xi — x + h (and x\ — x). If we let 
xq — x, x\ — x + h and X2 — x + 2h, then (9.152a) yields 

/ (1) 0O * ^-[-3/W + 4 f(x + ft) - f{x + 2ft)], (9.153) 

2n 

and if we let X2 — x, x\ — x — ft, and xq — x — 2ft, then (9.152c) yields 

f W (x) « i-[/(* - 2ft) - 4/(* - A) + 3/(*)]. (9.154) 

2ft 

Note that (9.153) and (9.154) were employed in obtaining Tcin) in (9.107). 
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PROBLEMS 
9.1. This problem is based on an assignment problem due to I. Leonard. Consider 



./o 



x + a 



■ dx, 
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where a > 1 and n e Z + . It is easy to see that for < x < 1, we have 
jc" +1 < x", and < I n +\ < I n for all n e Z + . For < x < 1, we have 



implying that 



or 



jc" x n x n 
< < — . 

1 + a x + a a 



f l x n f l x n 

I dx < I„ < / — dx 

h 1 +a Jo a 



1 1 

< In < 



(«+ l)(l+a) (n + l)o' 

so immediately lim n ^oo /„ = 0. Also, we have the difference equation 

f l x n ~ l [x + a-a] 1 

/„ = / dx = - - al n -\ (9.P.1) 

Jo x + a n 

for n e N, where 7 = /J ^ dx = [log e (x + a)], 1 , = log e (^)- 

(a) Assume that 7o = ^o + € is the computed value of Iq. Assume that no 
other errors arise in computing /„ for n > 1 using (9.P.1). Then 

1 

/„ = aI n -\. 

n 

Define the error e n — I n — I n , and find a difference equation for e„. 

(b) Solve for e n , and show that for large enough a we have linin^oo 
\e n \ = oo. 

(c) Find a stable algorithm to compute /„ for n e Z + . 

9.2. (a) Find an upper bound on the magnitude of the rounding error involved in 
applying the trapezoidal rule to 



Jo 



x dx. 



(b) Find an upper bound on the magnitude of the truncation error in applying 
the trapezoidal rule to the integral in (a) above. 

9.3. Consider the integral 

1(e) — I */xdx, 
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where a > e > 0. Write a MATLAB routine to fill in the following table for 
a — 1, and n — 100: 



e 


|7(e)-T(«)| B T(n) 


|7(e)- S(n)\ B s(n ) 


0.1000 
0.0100 
0.0010 
0.0001 







In this table T{n) is from (9.14), S(n) is from (9.41), and 

\Et(h)\ < Bt(ti), \Es(n)\ < Bs(n), 

where the upper bounds fir(n) and B${ n ) are obtained using (9.33) and (9.59), 
respectively. 



9.4. Consider the integral 



h i 



dx 



4' 



(a) Use the trapezoidal rule to estimate I, assuming that h = j, 

(b) Use Simpson's rule to estimate I, assuming that h — j. 

(c) Use the corrected trapezoidal rule to estimate I, assuming that h — j. 

Perform all computations using only a pocket calculator. 
9.5. Consider the integral 

•!/ 2 dx 



7-1/2 T 



In 3. 



(a) Use the trapezoidal rule to estimate I, assuming that h — j. 

(b) Use Simpson's rule to estimate I, assuming that h=\. 

(c) Use the corrected trapezoidal rule to estimate I, assuming that h — j. 

Perform all computations using only a pocket calculator. 
9.6. Consider the integral 

" n l 2 sin(3x) 



/ 

Jq 



Tt 

■ dx — — . 
smi 2 



(a) Use the trapezoidal rule to estimate I, assuming that h — ir/12. 

(b) Use Simpson's Rule to estimate I, assuming that h — tt/12. 

(c) Use the corrected trapezoidal rule to estimate I, assuming that h — n/Yl. 

Perform all computations using only a pocket calculator. 
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9.7. The length of the curve y — fix) for a < x < b is given by 

-b 



L= f Jl + [fVHx)] 2 dx. 

J a 



Suppose f{x) — cos*. Compute L for a — —jt/2 and b — jt/2. Use the 
trapezoidal rule, selecting n so that |£Y(n)l < 0.001. Hence, using (9.33), 
select n such that 

1 1 . 

^(b-afM < 0.001. 

12 n L 

Do the computations using a suitable MATLAB routine. 

9.8. Recall Example 6.5. Estimate the integral 

I — I e~ x dx 

by 

(a) Integrating the natural spline interpolant 

(b) Integrating the complete spline interpolant 

9.9. Find the constants a and /J in x — at + fi, and find / in terms of g such 
that 

f b f l fix) 

/ git)dt= I - dx. 

Ja J -i VI - x 1 

[Comment: This transformation will permit you to apply the Chebyshev- 
Gauss quadrature rule from Eq. (9.98) to general integrands.] 

rb 

9.10. Consider the midpoint rule (Section 9.2). If we approximate I — J a fix) dx 
by one rectangle, then the rule is 

' a + b 
RH) = ib-a)f 



2 
so the truncation error involved in this approximation is 



-i 



£«(!)= / fix)dx-Ril). 

J a 

Use Taylor expansion error analysis to find an approximation to Er(\). [Hint: 
With x = (a + b)/2 and h — b — a, we obtain 

fix) = fiJ) + (x- J)f W iJ) +^_(x- x) 2 f (2) (x) + ■■■■ 

Consider ./"(fl), f(b), and f fix)dx using this series expansion.] 
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9.11. Use both the trapezoidal rule [Eq. (9.14)] and the Chebyshev-Gauss quadra- 
ture rule [Eq. (9.98)] to approximate I — ^ Q ^^ dx for n — 6. Assuming 
that / = 1.1790, which rule gives an answer closer to this value for / ? Use 
a pocket calculator to do the computations. 



9.12. Consider the integral 



Jo 



cos x dx — — . 

2 



Write a MATLAB routine to approximate / using the trapezoidal rule, and 
Richardson's extrapolation. The program must allow you to fill in the fol- 
lowing table: 



n 


T(n) \I-T(n)\ 


R(n) \I-R(n)\ 


2 

4 

8 

16 

32 

64 
128 
256 
512 
1024 







Of course, the extrapolated values are obtained from (9.111), where I(n) = 
T(n). Does extrapolation improve on the accuracy? Comment on this. 

9.13. Write a MATLAB routine that allows you to make a table similar to that in 
Example 9.6, but for the integral 



Jo 



x l ' A dx. 



9.14. The complete elliptic integral of the first kind is 

^' 2 d6 



K(k) 



P7T/Z 

Jo vT 



,0 < k < 1, 



k 2 sin 2 
and the complete elliptic integral of the second kind is 

f*/2 

E(k)= VI -k 2 sm 2 6d6,0 < k < 1. 

Jo 

(a) Find a series expansion for K(k). [Hint: Recall (3.80).] 
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(b) Construct a Romberg table for N — 4 for the integral K(j). Use the 
series expansion from (a) to find the "exact" value of K(j) and compare. 

(c) Find a series expansion for E(k). [Hint: Recall (3.80).] 

(d) Construct a Romberg table for N — 4 for the integral E(j). Use the series 
expansion from (c) to find the "exact" value of E(j) and compare. 

Use MATLAB to do all of the calculations. [Comment: Elliptic integrals are 
important in electromagnetic potential problems (e.g., finding the magnetic 
vector potential of a circular current loop), and are important in analog and 
digital filter design (e.g., elliptic filters).] 

9.15. This problem is about an alternative approach to the derivation of Gauss-type 
quadrature rules. The problem statement is long, but the solution is short 
because so much information has been provided. Suppose that w(x) > for 
x e [a, b], so w(x) is some weighting function. We wish to find weights Wk 
and sample points Xk for k = 0, 1, . . . , n — 1 such that 



/ w(x)f(x)dx = J^w k f(x k ) (9.P.2) 

Ja , „ 



for all f(x) — x 1 , where j — 0, 1, . . . , In — 2, In — 1. This task is greatly 
aided by defining the moments 



-l 



b 

w(x)x } dx, 



where m ; is called the 7th moment of w(x). In everything that follows it is 
important to realize that because of (9.P.2) 



pb "-1 

ij = I x J w(x)dx = >•. 
Ja fc=0 



w k x J k . 



The method proposed here implicitly assumes that it is easy to find the 
moments m;. Once the weights and sample points are found, expression 
(9.P.2) forms a quadrature rule according to 

rb »-l 

/ w(x)f(x) dx » y^ Wkfjxk) = G{n - 1), 

Ja k=0 

where now f(x) is essentially arbitrary. Define the vectors 

w — [wqw\ ■ ■ ■ w n -2W n -\] e R", 
m = [m m\ ■ ■ ■ m 2 n-2m 2 n-\] T € R 2 ". 
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Find matrix A e R nx2n such that A T w — m. Matrix A turns out to be a 
rectangular Vandermonde matrix. If we knew the sample points Xk, then it 
would be possible to use A T w — m to solve for the weights in w. (This is 
so even though the linear system is overdetermined.) Define the polynomial 

n— 1 n 

Pn(x) — \\(X - Xj) = ^ p„,jX j 

;=o .7=0 

for which we see that p nn — 1. We observe that the zeros of p n (x) happen 
to be the sample points Xk that we are looking for. The following suggestion 
makes it possible to find the sample points Xk by first finding the polynomial 
p n (x). Using (in principle) Chapter 7 ideas, we can then find the roots of 
p n (x) — 0, and so find the sample points. Consider the expression 

n-\ 

mj+ r pn,j = y^WfcXft r Pn,j, (9.P.3) 

k=0 

where r — 0, 1, ...,« — 2, n — 1. Consider the sum of (9.P.3) over j — 
0, 1, . . . , n. Use this sum to show that 

n-i 

Y^mj+rPnj = -m„ +r (9.P.4) 

.7=0 

for r = 0, 1, . . . , n — 1. This can be expressed in matrix form as Mp — q, 
where 

M=[m i+i ] u=0 ,i,...,„-i€R" x " 

which is called a Hankel matrix, and where 

P = [Pn,0Pn,l ■ ■ ■ Pn,n-l] T € R" 

and 

-q = [m n m n+ \ ■ ■ ■ m 2 „-2»J2«-l] T '. 

Formal proof that M~ l always exists is possible, but is omitted from con- 
sideration. [Comment: The reader is warned that the approach to Gaussian 
quadrature suggested here is not really practical due to the ill-conditioning 
of the matrices involved (unless n is small).] 

9.16. In the previous problem let a = — 1, b = 1, and w(x) — Vl — x 2 . Develop a 
numerically reliable algorithm to compute the moments m ; - = f w(x)xi dx. 
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9.17. Derive Eq. (9.88). 

9.18. Repeat Example 9.4, except that the integral is now 



/ 



e x dx. 

1 



9.19. This problem is a preview of certain aspects of probability theory. It is an 
example of an application for numerical integration. An experiment produces 
a measurable quantity denoted x e R. The experiment is random in that the 
value of x is different from one experiment to the next, but the probability 
of x lying within a particular range of values is known to be 

P = P[ a < x < b] - / f x (x) dx, 



where 



1 / x 2 

fx(x) = exp - — 

V2tto- 2 \ 2<H 

which is an instance of the Gaussian function mentioned in Chapter 3. This 
fact may be interpreted as follows. Suppose that we perform the experiment R 
times, where R is a "large" number. Then on average we expect a < x < b 
a total of PR times. The function fx(x) is called a Gaussian probability 
density function (pdf) with a mean of zero, and variance a 2 . Recall Fig. 
3.6, which shows the effect of changing a 2 . In Monte Carlo simulations of 
digital communications systems, or for that matter any other system where 
randomness is an important factor, it is necessary to write programs that gen- 
erate simulated random variables such as x. The MATLAB randn function 
will generate zero mean Gaussian random variables with variance a 2 — 1. 
For example, x — randn(l, AO will load N Gaussian random variables into 
the row vector x. Write a MATLAB routine to generate N = 1000 simu- 
lated Gaussian random variables (also called Gaussian variates) using randn. 
Count the number of times x satisfies — 1 < x < 1. Let this count be denoted 
C. Your routine must also use the trapezoidal rule to estimate the probability 
P — P[—l < x < 1] using erf(x) [defined in Eq. (3.107)]. The magnitude of 
the error in computing P must be <0.0001. This will involve using the trun- 
cation error bound (9.33) in Chapter 9 to estimate the number of trapezoids 
n that you need to do this job. You are to neglect rounding error effects here. 
Compute C — PN. Your program must print C and C to a file. Of course, 
we expect C « C. 

9.20. Develop a MATLAB routine to fill in the following table, which uses the 
central difference approximation to the first derivative of a function [i.e., 
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fc (x)] to estimate f^\x), where here 

fix) = \og e x. 



h = 10" 4 h = 10" 5 h = 10" 6 h = 1(T 7 



10" 7 A = 10" 8 



f (1) (x) 



1.0 

2.0 
3.0 
4.0 
5.0 
6.0 
7.0 
8.0 
9.0 
10.0 



Explain the results you get. 

9.21. Suppose that f(x) = e _j; \ and recall (9.132). 

(a) Sketch f {2) (x). 

(b) Let h — 1/10, and compute f\ (1). Find an upper bound on |ey(l)|. 
Compute |/ (1) (1) - /) X) (1)| = |e/(l)|, and compare to the bound. 

(c) Let h = jq , and compute /S (1 / V2) . Find an upper bound on | e /■ ( 1 /-s/2) | . 

Compute |/W(l/-v/2) - /^'(l/v^)! = |e/(l/V2)|, and compare to the 
bound. 

9.22. Show that another approximation to f^\x) is given by 



1 



f w (x) w — -[8/(x + A) - 8/(x - h) - f(x + 2h) + f(x - 2ft)]. 
12ft 

Give an expression for the error involved in using this approximation. 
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10.1 INTRODUCTION 

In this chapter we consider numerical methods for the solution of ordinary differ- 
ential equations (ODEs). We recall that in such differential equations the function 
that we wish to solve for is in one independent variable. By contrast partial differ- 
ential equations (PDEs) involve solving for functions in two or more independent 
variables. The numerical solution of PDEs is a subject for a later chapter. 

With respect to the level of importance of the subject the reader knows that all 
dynamic systems with physical variables that change continuously over time (or 
space, or both) are described in terms of differential equations, and so form the 
basis for a substantial portion of engineering systems analysis, and design across 
all branches of engineering. The reader is also well aware of the fact that it is quite 
easy to arrive at differential equations that completely defy attempts at an analytical 
solution. This remains so in spite of the existence of quite advanced methods for 
analytical solution (e.g., symmetry methods that use esoteric ideas from Lie group 
theory [1]), and so the need for this chapter is not hard to justify. 

Where ODEs are concerned, differential equations arise within two broad cate- 
gories of problems: 

1. Initial-value problems (IVPs) 

2. Boundary-value problems (BVPs) 

In this chapter we shall restrict consideration to initial value problems. However, 
this is quite sufficient to accommodate much of electric/electronic circuit modeling, 
modeling the orbital dynamics of satellites around planetary bodies, and many other 
problems besides. 

A simple example of an ODE for which no general analytical theory of solution 
is known is the Duffing equation 



d x(t) dx(t) ■, 

m ^— + k — — + axit) + Sx 6 (t) = F cos(cot), (10.1) 

dt z dt 



An Introduction to Numerical Analysis for Electrical and Computer Engineers, by C.J. Zarowski 
ISBN 0-471-46737-5 © 2004 John Wiley & Sons, Inc. 
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where t > 0. Since we are concerned with initial-value problems we would need to 
know, at least implicitly, x(0), and x J- f which are the initial conditions. If it were 
the case that 5 = then the solution of (10.1) is straightforward because it is a 
particular case of a second-order linear ODE with constant coefficients. Perhaps the 
best method for solving (10.1) in this case would be the Laplace transform method. 
However, the case where 5^0 immediately precludes a straightforward analytical 
solution of this kind. It is worth noting that the Duffing equation models a forced 
nonlinear mechanical spring, where the restoring force of the spring is accounted 
for by the terms ax(t) + <Sx 3 (f). Function x(t), which we wish to solve for, is the 
displacement at time t of some point on the spring (e.g., the point mass m at the 
free end) with respect to a suitable reference frame. Term k-jp- is the opposing 
friction, while F cos(atf) is the periodic forcing function that drives the system. An 
example of a recent application for (10.1) is in the modeling of micromechanical 
filters/resonators [2]. 1 

At the outset we consider only first-order problems, specifically, how to solve 
(numerically) 

^- = f(x,t), x = x(0) (10.2) 

at 

for t > 0. [From now on we shall often write x instead of x(t), and dx/dt instead of 
dx(t)/dt for brevity.] However, the example of (10.1) is a second-order problem. 
But it is possible to replace it with a system of equivalent first-order problems. 
There are many ways to do this in principle. One way is to define 

y = £. do.3) 

dt 

The functions x(t) and y(t) are examples of state variables. Since we are interpret- 
ing (10.1) as the model for a mechanical system wherein x(t) is displacement, it 
therefore follows that we may interpret y(t) as velocity. From the definition (10.3) 
we may use (10.1) to write 

dy a k 8 -, F 

— = x y x -\ cos(oif) (10.4a) 

dt m m m m 

and 

dx 

— = y. (10.4b) 

dt 

The clock circuit in many present-day digital systems is built around a quartz crystal. Such crystals 
do not integrate onto chips. Micromechanical resonators are intended to replace the crystal since such 
resonators can be integrated onto chips. This is in furtherance of the goal of more compact electronic 
systems. This applications example is a good illustration of the rapidly growing trend to integrate 
nonelectrical/nonelectronic systems onto chips. The implication of this is that it is now very necessary 
for the average electrical and/or computer engineer to become very knowledgeable about most other 
branches of engineering, and to possess a much broader and deeper knowledge of science (physics, 
chemistry, biology, etc.) and mathematics. 
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Equations (10.4) have the forms 



dx 

— = f(x,y,t) 

dt 

dy 

— =g(x,y,t) 

dt 



(10.5a) 



(10.5b) 



for the appropriate choices of / and g. The initial conditions for our example 
are x(0) and y(0) (initial position and initial velocity, respectively). These rep- 
resent a coupled system of first-order ODEs. Methods applicable to the solution 
of (10.2) are extendable to the larger problem of solving systems of first-order 
ODEs, and so in this way higher-order ODEs may be solved. Thus, we shall also 
consider the numerical solution of initial-value problems in systems of first-order 
ODEs. 

The next two examples illustrate how to arrive at coupled systems of first-order 
ODEs for electrical and electronic circuits. 

Example 10.1 Consider the linear electric circuit shown in Fig. 10.1. The input 
to the circuit is the voltage source v s (t), while we may regard the output as the 
voltage drop across capacitor C, denoted vc(t). The differential equation relating 
the input voltage v s (t) and the output voltage vc (t) is thus 



LiC 



d 3 v C (t) 
dt 3 



R\C 



d 2 v c (t) 
dt 2 






dvcif) 
dt 



R\ 
—vc(t) 

i>2 



dv s {t) 
dt 



(10.6) 



This third-order ODE may be obtained by mesh analysis of the circuit. The reader 
ought to attempt this derivation as an exercise. One way to replace (10.6) with 
a coupled system of first-order ODEs is to define the state variables Xk(t) (k e 
{0, 1, 2}) according to 



xo(t) = v c (t), 



xi(t) = 



dv C (t) 
dt 



Substituting (10.7) into (10.6) yields 



xi{t) = 



d 2 v c (t) 
dt 2 ' 



(10.7) 



LiC— f^ + RiCx 2 (t) + 7 + 1 J|(0+ 7^*0(0 = -^r 1 - 

dt \L2 J L2 dt 





Ri 


<-i 




L 2 








+ \ 




'l,W 


c 















Figure 10.1 The linear electric circuit for Example 10.1. 
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If we recognize that 



xi (t) = 



dx(j(t) 
dt 



x 2 (t) = 



dx\(t) 
dt 



then the complete system of first order ODEs is 

dx (t) 



dt 










dxi(t) 

—, — =x 2 (t), 
dt 








(10.8) 


dx2(t) 1 dv s (t) 
dt L\C dt 


R\ 
- —x 2 (t) - 


LiC \L 2 J 


xi(t)- 


R 

- T j r xo(t)- 
LIL2C 



In many ways this is not the best description for the circuit dynamics. 

Instead, we may find the matrix A e R 3x3 and column vector b e R 3 such that 



dvc(t) 

dt 
dJL { (t) 

dt 

dJL 2 (t) 

dt 



= A 



v c (t) 



bv s {t). 



(10.9) 



This defines a new set of state equations in terms of the new state variables Vc(t), 
JLj(?)> an d 'l 2 (0- The matrix A and vector b contain constants that depend only 
on the circuit parameters R\,L\, L2, and C. 

Equation (10.9) is often a better representation than (10.8) because 

1. There is no derivative of the forcing function v s (t) in (10.9) as there is in 
(10.8). 

2. There is a general (linear) theory of solution to (10.9) that is in practice easy 
to apply, and it is based on state-space methods. 

3. Inductor currents [i.e., iz,[(f), *'l, (?)] and capacitor voltages [i.e., vc(t)] can 
be readily measured in a laboratory setting while derivatives of these are 
not as easily measured. Thus, it is relatively easy to compare theoretical and 
numerical solutions to (10.9) with laboratory experimental results. 



Since 



kit) = c 



dv c (t) 
dt 



VL,(t) = Li 



dJL, (t) 
dt 



v L Jt) = L 



dJL 2 (t) 
dt 



on applying Kirchoff's Voltage law (KVL) and Kirchoff's Current law (KCL), we 
arrive at the relevant state equations as follows. 
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First 

v s (t) = RiiLi(t) + Li 

We see that i>c(t) — Vi 2 (t), and so 

vc(t) = i 



dJLi(t) 
dt 



ih 2 {t) 

i . 

dt 



L 2 



dJL 2 (t) 
dt 



(10.10) 



giving 



dJL 2 (t) 
dt 



1 
= — v c (t), 

^2 



(10.11) 



which is one of the required state equations. Since 

MO = *'c(0 + MO, 



and so 



we also have 



dvr(t) 
M0 = C— r^ + MO, 



dt 



1 



dvc(t) 1 

~dT = c iL > (t) - c iL ^ ty 



(10.12) 



This is another of the required state equations. Substituting (10.11) into (10.10) 
gives the final state equation 



di Ll (t) 



R 



l . 



MO- 



l 



-v c (t) 



1 



-V s (t). 



dt L\ "' ' L\ ' L\ 

The state equations may be collected together in matrix form as required: 

dvc(t) 



(10.13) 



dt 
diL, (0 

dt 
diL 2 (t) 

dt 






l 


1 "I 


c 


c 


1 


R\ 





~L\ 


U 


1 

_ T 2 









vc(0 
MO 
MO 



o 

i 





i^(0- 



=/i 



Example 10.2 Now let us consider a more complicated third-order nonlinear 
electronic circuit called the Colpitts oscillator [9]. The electronic circuit, and its 
electric circuit equivalent (model) appears in Fig. 10.2. This circuit is a popular 
analog signal generator with a long history (it used to be built using vacuum 
tubes). The device Q is a three-terminal device called an NPN-type bipolar junction 
transistor (BJT). The detailed theory of operation of BJTs is beyond the scope of 
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v cc( f )C) v cc(t)i 





(a) V EE (b) V EE - 

Figure 10.2 The BJT Colpitts oscillator (a) and its electric circuit equivalent (b). 



this book, but may be found in basic electronics texts [10]. For present purposes 
it is enough to know that Q may be represented with a nonlinear resistor R (the 
resistor enclosed in the box in Fig. 10.2b), and a current-controlled current source 
(CCCS), where 

ic(t) = Mb(1) (10.14) 



and 



*'fl00 = 



0, v BE {t)<V TH 

(vbe(i) - V T h) 



Ron 



vbeQ) > Vth 



(10.15) 



The current is (?) is called the base current of Q, and flows into the base terminal 
of the transistor as shown. From (10.15) we observe that if the base-emitter voltage 
vbe(i) is below a threshold voltage Vth, then there is no base current into Q (i.e., 
the device is cut off). The relationship between vbe( 1 ) — Vth, an d *b(0 obeys 
Ohm's law only when VBE(t) is above threshold, in which case Q is active. In 
either case (10.14) says the collector current ic(t) is directly proportional to /#(?), 
and the constant of proportionality f>p is called the forward current gain of Q. 
Voltage vcE(t) is the collector-emitter voltage of Q. Typically, Vth ^ 0.75 V, j3p 
is about 100 (order of magnitude), and Ron (on resistance of Q) is seldom more 
than hundreds of ohms in size. 
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From (10.15) is it) is a nonlinear function of vgEit) that we may compactly write 
as isit) = fitivBEit))- There are power supply voltages vccit), and Vee- Voltage 
Vee < is a constant, with a typical value Vee — —5 V. Here we treat vccit) 
as time- varying, but it is usually the case that (approximately) vccit) = Vccuit), 
where 

«(*) = { J; [H ■ ao.16) 

Function u(t) is the unit step function. To say that vcc it) — Vcc u it) is to say that 
the circuit is turned on at time t — 0. Typically, Vcc — +5 V. 

The reader may verify (again as a circuit analysis review exercise) that state 
equations for the Colpitts oscillator are: 

c ^dvcEit)_ = ,^ f) _ p pfR{vBE{t)) ^ (10.17a) 

at 

dVRFit) VtjF it) + Vff 
Cl-^ 1 = -- -^i — - fRiVBEit)) ~ iLit), (10.17b) 

dt R E e 

L dnM = vcc _ vcE{t) + _ RtiL(t) (10.17c) 

dt 

Thus, the state variables are vbe^), vcEit), and iiit)- As previously, this circuit 
description is not unique, but it is convenient. 

Since numerical methods only provide approximate solutions to ODEs, we are 
naturally concerned about the accuracy of these approximations. There are also 
issues about the stability of proposed methods, and so this matter as well will be 
considered in this chapter. 



10.2 FIRST-ORDER ODEs 

Strictly speaking, before applying a numerical method to the solution of an ODE, 
we must be certain that a solution exists. We are also interested in whether the 
solution is unique. It is worth stating that in many cases, since ODEs are often 
derived from problems in the physical world, existence and uniqueness are often 
"obvious" for physical reasons. Notwithstanding this, a mathematical statement 
about existence and uniqueness is worthwhile. 

The following definition is needed by the succeeding theorem regarding the 
existence and uniqueness of solutions to first order ODE initial value problems. 

Definition 10.1: The Lipschitz Condition The function fix, t) & R satisfies 
a Lipschitz condition in x for S C R 2 iff there is an a > such that 

\f(x,t)-f(y,t)\<a\x-y\ 

when (x, t), (y,t) e S. The constant a is called a Lipschitz constant for f(x,t). 
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It is apparent that if f(x, t) satisfies a Lipschitz condition, then it is smooth in 
some sense. The following theorem is about the existence and uniqueness of the 
solution to 

^ = /(*,*) (10.18) 

at 

for < t < tf, with the initial condition xo — x(0). Time t — is the initial time, or 
starting time. We call constant tf the final time. Essentially, we are only interested 
in the solution over a finite time interval. This constraint on the theory is not 
unreasonable since a computer can run for only a finite amount of time anyway. 
We also remark that interpreting the independent variable t as time is common 
practice, but not mandatory in general. 

Theorem 10.1: Picard's Theorem Suppose that S — {{x, f) € R 2 |0 < t < 
tf, — oo < x < oo}, and that f(x, t) is continuous on S. If / satisfies a Lipschitz 
condition on set S in the variable x, then the initial- value problem (10.18) has a 
unique solution x — x(t) for all < t < tf. 

Proof Omitted. We simply mention that it is based on the Banach fixed-point 
theorem (recall Theorem 7.3). 

We also mention that a proof of a somewhat different version of this theorem 
appears in Kreyszig [3, pp. 315-317]. It involves working with a contractive map- 
ping on a certain closed subspace of C(J), where / = [to — fi, to + /J] C R and 
C(J) is the metric space of continuous functions on /, where the metric is that 
of (1.8) in Chapter 1. It was remarked in Chapter 3 [see Eq. (3.8)] that this space 
is complete. Thus, any closed subspace of it is complete as well (a fact that was 
mentioned in Chapter 7 following Corollary 7.1). 

We may now consider specific numerical techniques. Define x„ — x(t n ) for 
n e Z + . Usually we assume that to — 0, and that 

t n +i=t n +h, (10.19) 

where h > 0, and we call h the step size. From (10.18) 

4 l) = ^r = ^P-\t= tn = /(*„, t n ). (10.20) 

at at 

We may expand solution x(t) in a Taylor series about t — t„. Therefore, since 
x(t n+ \) = x(t n +h) = x n+ \, and with xj, ) = x (k) (t n ) 

x n+1 =x n + hx^ + ^h 2 xj?> + ^h 3 4 3) + ■■■■ (10-21) 

(k) 

If we drop terms in x„ f or k > 1, then (10.21) and (10.20) imply 



*•«+! — %n ~r ^^n — %n \ *V \^n-> *n)' 
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Since xq — x(to) — x(0) we may find (x„) via 

x„+i — x„ + hf(x n , t n ). (10.22) 

This is often called the Euler method (or Euler's method)? A more accurate descrip- 
tion would be to call it the explicit form of Euler's method in order to distinguish 
it from the implicit form to be considered a little later on. The distinction matters 
in practice because implicit methods tend to be stable, whereas explicit methods 
are often prone to instability. 

A few general words about stability and accuracy are now appropriate. In what 
follows we will assume (unless otherwise noted) that the solution to a differen- 
tial equation remains bounded; that is, \x{t)\ < M < oo for all t > 0. However, 
approximations to this solution [e.g., (x n ) from (10.22)] will not necessarily remain 
bounded in the limit as n — > oo; that is, our numerical methods might not always be 
stable. Of course, in a situation like this the numerical solution will deviate greatly 
from the correct solution, and this is simply unacceptable. It therefore follows that 
we must find methods to test the stability of a proposed numerical solution. Some 
informal definitions relating to stability are 

Stable method: The numerical solution does not grow without bound (i.e., "blow 
up") with any choice of parameters such as step size. 

Unstable method: The numerical solution blows up with any choices of param- 
eters (such as step size). 

Conditionally stable method: For certain choices of parameters the numerical 
solution remains bounded. 

We mention that even if the Euler method is stable, its accuracy is low because 
only the first two terms in the Taylor series are retained. More specifically, we 
say that it is a first-order method because only the first power of h is retained in 
the Taylor approximation that gave rise to it. The omission of higher-order terms 
causes truncation errors. Since h 2 (and higher power) terms are omitted we also 
say that the truncation error per step (sometimes called the order of accuracy) is 
of order h 2 . This is often written as 0(h 2 ). (Here we follow the terminology in 
Kreyszig [4, pp. 793-794].) In summary, we prefer methods that are both stable, 
and accurate. It is important to emphasize that accuracy and stability are distinct 
concepts, and so must never be confused. 

Strictly speaking, in truncating the series in (10.21) we should write x nJr \ = x n + hf(x n , t n ) so that 
Euler's method is 

x n+ i =x„ +hf(x„,t„) 

with xq = xq. This is to emphasize that the method only generates approximations to x„ =x(t n ). 
However, this kind of notation is seldom applied. It is assumed that the reader knows that the numerical 
method only approximates x(t n ) even though the notation does not necessarily explicitly distinguish 
the exact value from the approximate. 
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We can say more about the accuracy of the Euler method: 

Theorem 10.2: For dx(t)/dt — f(x(t), t) let f(x(t), t) be Lipschitz continu- 
ous with constant a (Definition 10.1), and assume that x(t) e C 2 [?o, tf] (tf > to). 
If x n s» x(t n ) — x n , where (t n — to + nh, and t n < tf) 

Xn+\ = X n + hf(x n , t n ), (XO « x(t )) 

then 

1 e a(t„-t ) _ i 

\x(t„) -x n \< e a(t »-' o) \x(to) - xq\ + -hM , (10.23) 

2 a 

where M = mzx, €[tQaf ] |x (2) (f)|. 
Proof Euler' s method is 

Xn+\ — X n -\- rlj \X n , t n ) 

and from Taylor's theorem 

x(t n+ i) = x(t n ) + hx (l) (t n ) + \h 2 x^^ n ) 
for some £„ € [t n , t n+ {\. Thus 

x(t n +\) ~ X n +\ = X(t„) - X n + h[x (l) (t n ) - f(x n , t n )] + \h 2 X {2) (£„) 

= X(t n ) -X„ + h[f(x(t„), t n ) - f(x„, t n )] + \h 2 X {2 \^ n ) 

so that 

\x(t„+i) - x n +l\ < \x{t n ) -x„\ +ah\x(t„) -x„\ + \h 2 \x {2) (%„)\. 

For convenience we will let e n — \x(t n ) — x„\, X — 1 + ah, and r„ — jh 2 x^ 2 \^ n ), 
so that 

e«+l < ^e n +r n . 

It is easy to see that 3 

e\ < Xe Q + >"o 

£2 < Xe\ +r\ — X 2 eo + Xro + r\ 

e3 < Xe2 + V2 — X 3 eo + X 2 ro + Xr\ + r2 



n-\ 



e n < X n e Q + ^2x'r n -i-j 



More formally, we may use mathematical induction. 
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If M — max t€ y ^ f ] \x^(t)\ then r n -\-j < jh 2 M, and hence 



n-\ 



e n < X n e + jh 2 Mj2 kJ ' 



and since J2"j=o ^'' = T^T ' anc * f°r x > — 1 we nave (1 + x) n < e " x ', thus 

«zi . k n _ l £ nah _ j g afe-r ) _ { 

> !■' = < = . 

*-^ X — 1 ah ah 

j=o 

Consequently, 

2 e a(t n -to) _ 1 

e„ < e afe -' o) e + -AM 

2 a 

which immediately yields the theorem statement. 

We remark that eo = \x(to) — xq\ — only if xo — x(to) exactly. Where quan- 
tization errors (recall Chapter 2) are concerned, this will seldom be the case. The 
second term in the bound of (10.23) may be large even if h is tiny. In other words, 
Euler's method is not necessarily very accurate. Certainly, from (10.23) we can 
say that e n oc h. 

As a brief digression, we also note that Theorem 10.2 needed the bound 

(l+x) n < e nx (x > -1). (10.24) 

We may easily establish (10.24) as follows. From the Maclaurin expansion 
(Chapter 3) 

e x = 1 + x + \x 2 e$ 
(for some % e [0, x]) so that 

< 1 + x < l+x + ±x V = e x , 
and because 1 + x > (i.e., x > — 1) 

0< (l+x)" <e nx , 
thus establishing (10.24). 
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The stability of any method may be analyzed in the following manner. First 
recall the Taylor series expansion of fix, t) about the point (xq, to) 



tt *\ ti t \< n t \ d f( x °' fo) , < df(x ,t ) 

fix, t) = fixo, to) + it - to) \-ix- xo)- 



1 
2! 



it - to) 



dt 

2 d 2 fix , to) 



dx 



dt 2 



+ 20 - to)ix - xo) 



d Z fixo, tp) 
dtdx 



+(x — xq) 



2 d 2 fjx , to) 
dx 2 



(10.25) 



If we retain only the linear terms of (10.25) and substitute these into (10.18), then 
we obtain 



(i),,, dx t s,t. . .dfix^to) dfixp, t ) 

x 'it) = — = fixo, to) + it - t ) \-ix- x Q ) 

dt dt ox 



dfixo, to) , dfixo, to) 

■ x-\ 1- 



dx 



dt 



,, N dfixo, to) dfixo, t ) 

f ixo, t ) - t xq- 



dt 



=/., 



= A2 



so this has the general form (with k, k\, and X2 as constants) 



dx 



(10.26) 



dx 

— = Xx + k\t + A.2. 

dt 



(10.27) 



This linearized approximation to the original problem in (10.18) allows us to inves- 
tigate the behavior of the solution in close proximity to (xo, to). Equation (10.27) 
is often simplified still further by considering what is called the model problem 



dx 
dt 



(10.28) 



Thus, here we assume that k\t + k 2 in (10.27) can also be neglected. However, 
we do remark that (10.27) has the form 



dx 
dt 



+ Pit)x = Qit)x n , 



(10.29) 



where n = 0, Pit) — —X, and Qit) — k\t + X 2 - Thus, (10.27) is an instance of 
Bernoulli's differential equation [5, p. 62] for which a general method of solution 
exists. But for the purpose of stability analysis it turns out to be enough (usually, 
but not always) to consider only (10.28). Equation (10.28) is certainly simple in 
that its solution is 

x(t) = xiO)e Xt . (10.30) 
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If Euler's method is applied to (10.28), then 

x n+ i — x„ + hXx n = (1 + hX)x n . (10.31) 

Clearly, for n e Z + 

Xn = (1 + hX)"x , (10.32) 

and we may avoid lirrin^oo \x n \ — oo if 

|l + Wi| < 1. 

The model problem (10.28) with the solution (10.30) is stable 4 only if X < 0. 
Hence Euler's method is conditionally stable for 

2 
X < and h < — , (10.33) 

- \X\ 

and is unstable if 

\l + hX\ > 1. (10.34) 

We see that depending on X and h, the explicit Euler method might be unstable. 

Now we consider the alternative implicit form of Euler's method. This method 
is also called the backward Euler method. Instead of (10.22) we use 

Xn + l = X„ + hf(x n +\, t n +l). (10.35) 

It can be seen that a drawback of this method is the necessity to solve (10.35) for 
x n+ \. This is generally a nonlinear problem requiring the techniques of Chapter 7. 
However, a strength of the implicit method is enhanced stability. This may be 
easily seen as follows. Apply (10.35) to the model problem (10.28), yielding 

x n + \ — x n + Xhx n -\-l 

or 

1 

x„+i = —x n . (10.36) 

1 — Xn 



Clearly 

1 



1 -Xh 



x . (10.37) 



Since we must assume as before that X < 0, the backward Euler method is stable 
for all h > 0. In this sense we may say that the backward Euler method is uncon- 
ditionally stable. Thus, the implicit Euler method (10.35) is certainly more stable 

For stability we usually insist that X < as opposed to allowing X = 0. This is to accommodate a 
concept called bounded-input, bounded-output (BIBO) stability. However, we do not consider the details 
of this matter here. 
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than the previous explicit form (10.22). However, the implicit and explicit forms 
have the same accuracy as both are first-order methods. 

Example 10.3 We wish to apply the implicit and explicit forms of the Euler 
method to 

dx 

— +x = 

dt 

for x(0) = 1. Of course, since this is a simple linear problem, we immediately 
know that 

x(t) = x(0)e~' = e~' 

for all t > 0. Since we have f(x, t) — —x, we obtain 

df(x,t) = 1 
dx 

implying that X — — 1. Thus, the explicit Euler method (10.22) gives 

x n+ i — (1 — h)x n (10.38a) 

for which < h < 2 via (10.33). Similarly, (10.35) gives for the implicit method 



1 

x n +i = rx n (10.38b) 

1 + h 

for which h > 0. In both (10.38a) and (10.38b) we have x Q = 1. 

Some typical simulation results for (10.38) appear in Fig. 10.3. Note the insta- 
bility of the explicit method for the case where h > 2. 

It is to be noted that a small step size h is desirable to achieve good accuracy. 
Yet a larger h is desirable to minimize the amount of computation involved in 
simulating the differential equation over the desired time interval. 

Example 10.4 Now consider the ODE 

dx _ t 2 3 

h 2tx — te x 

dt 

[5, pp. 62-63]. The exact solution to this differential equation is 



,2, 



3 



x(t) = T~ 2t , (10-39) 

e ' + ce a 
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Figure 10.3 Illustration of the implicit and explicit forms of the Euler method for the 
differential equation in Example 10.3. In plot (a), h is small enough that the explicit method 
is stable. Here the implicit and explicit methods display similar accuracies. In plot (b), h 
is too big for the explicit method to be stable. Instability is indicated by the oscillatory 
behavior of the method and the growing amplitude of the oscillations with time. However, 
the implicit method remains stable, but because h is quite large, the accuracy is not very 
good. 



for t > 0, where c = -^ — 1, and we assume that c > 0. Thus 



f(x,t) = t( 



-t l Ji 



Itx 



so 



df(x, t) .2 9 

= 3te ' x 2 -It. 
dx 



Consequently 



_ df(x ,t ) _ 8/(*o,0) _ n 

A — : — : — u. 



dx 



dx 



Via (10.33) we conclude that h > is possible for both forms of the Euler method. 
Since stability is therefore not a problem here, we choose to simulate the differential 
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Figure 10.4 Illustration of the explicit Euler method for the differential equation in 
Example 10.4. Clearly, although stability is not a problem here, the accuracy of the method 
is better for smaller h. 



equation using the explicit Euler method as this is much simpler to implement. 
Thus, from (10.22), we obtain 



ti ..3 



Xn-\-l — %n T" <l\tfi€ n ' X n Ll n X n \. 



(10.40) 



We shall assume xo = 1 [initial condition x(0)]. Of course, t n — hn for n — 
0, 1,2,.... 

Figure 10.4 illustrates the exact solution from (10.39), and the simulated solution 
via (10.40) for h — 0.02 and h — 0.20. As expected, the result for h — 0.02 is more 
accurate. 

The next example involves an ODE whose solution does not remain bounded 
over time. Nevertheless, our methods are applicable since we terminate the simu- 
lation after a finite time. 



Example 10.5 Consider the ODE 

dx 
dt 



— t 



2x 
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for t > to > [5, pp. 60-61]. The exact solution is given by 

,3 



x(t) 



f 



c 
572- 



(10.41) 



The initial condition is x(to) = xo with to > 0, and so 

,3 



v ? 



5?, 







implying that 



c = 5fg.ro - fj. 



2.t 



Since f(x,t) = t—=f-,we have 



A.= 



9/C*o, fo) 

3 A' 



2 
to' 



so via (10.33) for the explicit Euler method 

< h < to. 

However, this result is misleading here because x(t) is not bounded with time. In 
other words, it does not really apply here. From (10.22) 



X n +\ — X„ + h 



Zx n 
tn _ 



(10.42a) 



where 



t n — to + nh 
for n e Z + . If we consider the implicit method, then via (10.35) 

2 2x n -|-l 



Xn + 1 



h 



'm + 1 



tn + 1 . 



*n+l — 



ht 2 ,, 
n + \ 



2h 

tn + l 



(10.42b) 



where t n+ \ — to + (n + \)h for n e Z + . 

Figure 10.5 illustrates the exact solution x(t) from (10.41) along with the simu- 
lated solutions from (10.42). This is for x Q — 1 and t — 0.05 with h — 0.025 (a), 
and h — 5 (b). It can be seen that the implicit method is more accurate for t close 
to fo- Of course, this could be very significant since startup transients are often of 
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Figure 10.5 Illustration of the explicit and implicit forms of the Euler method for the ODE 
in Example 10.5. In plot (a), note that the implicit form tracks the true solution x(t) better 
near fy = 0.05. In plot (b), note that both forms display a similar accuracy even though h 
is huge, provided t ^> t$. 



interest in simulations of dynamic systems. It is noteworthy that both implicit and 
explicit forms simulate the true solution with similar accuracy for t much bigger 
than to even when h is large. 

This example is of a "stiff system" (see Section 10.6). 

Example 10.5 illustrates that stability and accuracy issues with respect to the 
numerical solution of ODE initial-value problems can be more subtle than our 
previous analysis would suggest. The reader is therefore duly cautioned about 
these matters. 

Recalling (3.71) from Chapter 3 (or recalling Theorem 10.2), the Taylor formula 
for x(t) about t = t n is [recall x n = x(t n ) for all n] 



l n + l 



X n +hf(x n ,t n )+-h 2 X (2) (i;) 

X — v- J £• 



(10.43) 



=J£n+l 



for some £ € [?„, t n+ \]. Thus, the truncation error per step in the Euler method is 
defined to be 

e„+l = x n+l - x n+l = \h 2 x { ^{^). (10.44) 
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We may therefore state, as suggested earlier, that the truncation error per step is of 
order 0(h 2 ) because of this. The usefulness of (10.44) is somewhat limited in that 
it depends on the solution x(t) [or rather on the second derivative x^ 2 '(f)], which 
is, of course, something we seldom know in practice. 

How may we obtain more accurate methods? More specifically, this means 
finding methods for which the truncation error per step is of order 0(h m ) with 
m > 2. 

One way to obtain improved accuracy is to try to improve the Euler method. 
More than one possibility for improvement exists. However, a popular approach is 
Heun 's method. It is based on the following observation. A drawback of the Euler 
method in (10.22) is that f(x n , t„) is the derivative x^(t) at the beginning of the 
interval [t n , f n +i]> ar) d yet x^(t') varies over [t n , t n+ {\. The implicit form of the 
Euler method works with f(x n +\, t n +\), namely, the derivative at t — t n+ \, and so 
has a similar defect. Therefore, intuitively, we may believe that we can improve 
the algorithm by replacing f(x n , t n ) with the average derivative 

\ [f(Xn, t n ) + f(x„ + hf(x n ,t„), t n + K)] . (10.45) 

This is approximately the average of x^'(f) at the endpoints of interval [t n , t n +{\. 
The approximation is due to the fact that 

/On+l, * B +i) ^ /(•*« + hf(x n , t n ), t n + h). (10.46) 

We see in (10.46) that we have employed (10.22) to approximate x n +\ according 
to x„ + i — x n + hf(x n , t n ) (explicit Euler method). Of course, t n+ \ — t„ + h does 
not involve any approximation. Thus, Heun 's method is defined by 

x n +i — x n + - [f(x n , t n ) + f(x„ + hf(x n , f„), t n + h)] . (10.47) 

This is intended to replace (10.22) and (10.35). 

However, (10.47) is an explicit method, and so we may wonder about its sta- 
bility. If we apply the model problem to (10.47), we obtain 

x n+1 = |"l+Aii+ I/j 2 l 2 ljt„ (10.48) 



for which 



[l+l/! + i/j 2 A 2 l\ - (10.49) 



For stability we must select h such that we avoid linin^oo \x n \ — oo. For conve- 
nience, define 

a = \+Xh + \h 2 X 2 , (10.50) 

so this requirement implies that we must have \a\ < 1. A plot of (10.50) in terms 
of hX appears in Fig. 10.6. This makes it easy to see that we must have 

-2<hX<0. (10.51) 
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Figure 10.6 A plot of a in terms of hX [see Eq. (10.50)]. 



Since X < is assumed [again because we assume that x(t) is bounded] (10.51), 
implies that Heun's method is conditionally stable for the same conditions as in 
(10.33). Thus, the stability characteristics of the method are identical to those of 
the explicit Euler method, which is perhaps not such a surprize. 

What about the accuracy of Heun's method? Here we may see that there is an 
improvement. Again via (3.71) from Chapter 3 



X n +\ 



K n + hx (l) (t n ) + ±h 2 X (2 Ht n ) + ±/*V 3 >(£) 



(10.52) 



for some f e \t n , t n +i]. We may approximate x^ 2 \t„) using a. forward difference 
operation 

x (l Ht n+l )-x^(t n ) 



x (2 \t n ) 



h 



(10.53) 



so that (10.52) becomes [using x^(t n+ i) — f(x n+ i,t n+ i), and x"'{t n ) 

f{Xn,t n )] 



x n + l 



1 2 

hf(x„,t„)+ -h 



X^{t n+ i)-X^{tn) 



(1)/ 



+ -/z 3 x (3) (f), (10.54) 
6 



or upon simplifying this, we have 



Xn+l = X n + - [f(x n , t n ) + f(x n +\, t n +l)] + -h 3 X°\%). 



(10.55) 



Replacing f(x n +\, f n +i) in (10.55) with the approximation (10.46), and dropping 
the error term, we see that what remains is identical to (10.47), namely, Heun's 
method. Various approximations were made to arrive at this conclusion, but they 
are certainly reasonable, and so we claim that the truncation error per step for 
Heun's method is 

e n+l = i/*V 3 >(f), (10.56) 
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where again f € [t n , f„+i], and so this error is of the order 0(h 3 ). In other words, 
although Heun's method is based on modifying the explicit Euler method, the 
modification has lead to a method with improved accuracy. 

Example 10.6 Here we repeat Example 10.5 by applying Heun's method under 
the same conditions as for Fig. 10.5a. Thus, the differential equation is again 



dx 

dt 



2x 

t 



and again we choose xo = 1.0, ?o = 0.05, with h — 0.025. The simulation result 
appears in Fig. 10.7. 

It is very clear that Heun's method is distinctly more accurate than the Euler 
method, especially near t — frj- 



Heun's method may be viewed in a different light by considering the following. 
We may formally integrate (10.18) to arrive at x(t) according to 



x{t) - x(t n ) 



f 



f(x, x)dx. 



(10.57) 
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Figure 10.7 Comparison of the implicit Euler method with Heun's method for 
Example 10.6. We see that Heun's method is more accurate, especially for t near fg = 0.05. 
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So if t = t n +\, then, from (10.57), we obtain 



n+1 

x(t„+i) = x(t„) + I f(x,x)dx 






or 



r'n+i 
x„ + i=x„+/ f(x, x) dr. (10.58) 

According to the trapezoidal rule for numerical integration (see Chapter 9, 
Section 9.2), we have 

/"»+' h 

/ f(x, x)dx «s -[fix„,t„) + fix n+ i,t n+ i)] 

(since h — t n +\ — t n ), and hence (10.58) becomes 

h 

X n +1 — X n + ~[fix„, t n ) + fiXn+l, t„+l)], (10.59) 

which is just the first step in the derivation of Heun's method. We may certainly 
call (10.59) the trapezoidal method. Clearly, it is an implicit method since we must 
solve (10.59) for x n+ \. Equally clearly, Eq. (10.59) appears in (10.55), and so we 
immediately conclude that the trapezoidal method is a second-order method with 
a truncation error per step of order 0(/z 3 ). Thus, we may regard Heun's method 
as the explicit form of the trapezoidal method. Or, equivalently, the trapezoidal 
method can be regarded as the implicit form of Heun's method. We mention that 
the trapezoidal method is unconditionally stable, but will not prove this here. 

The following example illustrates some more subtle issues relating to the stability 
of numerical solutions to ODE initial-value problems. It is an applications example 
from population dynamics, but the issues it raises are more broadly applicable. The 
example is taken from Beltrami [6]. 

Example 10.7 Suppose that x(t) is the total size of a population (people, 
insects, bacteria, etc.). The members of the population exist in a habitat that can 
realistically support not more than N individuals. This is the carrying capacity 
for the system. The population may grow at some rate that diminishes to zero 
as x(f) approaches N. But if the population size x(f) is much smaller than the 
carrying capacity, the rate of growth might be considered proportional to the present 
population size. Consequently, a model for population growth might be 



dxit) 
dt 



1 xjt) 

N 



(10.60) 



This is called the logistic equation. By separation of variables this equation has 
solution 

xit) = N n (10.61) 

1 + ce " 
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for t > 0. As usual, c depends on the initial condition (initial population size) x(0). 
The exact solution in (10.61) is clearly "well behaved." Therefore, any numerical 
solution to (10.60) must also be well behaved. 

Suppose that we attempt to simulate (10.60) numerically using the explicit Euler 
method. In this case we obtain [via (10.22)] 

x n +\ = (1 + hr)x„ - -Tj-xl. (10.62) 

Suppose that we transform variables according to x„ — ay n in which case (10.62) 
can be rewritten as 



y n+ \ — (1 + hr)y n 
If we select 



lira 

1 Vn 

N{\ + hrY " 



(10.63) 



then (10.63) becomes 



N(l + hr) 
hr 



y n+l = ky n {\ - y n ), (10.64) 



where A. = l+hr. We recognize this as the logistic map from Chapter 7 [see 
Examples 7.3-7.5 and Eq. (7.83)]. From Section 7.6 in particular we recall that 
this map can become chaotic for certain choices of X. In other words, chaotic 
instability is another possible failure mode for a numerical method that purports to 
solve ODEs. 

The explicit form of the Euler method is actually an example of a first-order 
Runge-Kutta method. Similarly, Heun's method is an example of a second-order 
Runge-Kutta method. It is second-order essentially because the approximation 
involved retains the term in h 2 in the Taylor series expansion. We mention that 
methods of still higher order can be obtained simply by retaining more terms in the 
Taylor series expansion of (10.21). This is seldom done because to do so requires 
working with derivatives of increasing order, and this requires much computational 
effort. But this effort can be completely avoided by developing Runge-Kutta meth- 
ods of higher order. We now outline a general approach for doing this. It is based 
on material from Rao [7]. 

All Runge-Kutta methods have a particular form that may be stated as 

x n +i — x„ +ha{x n , t n , h), (10.65) 

where a(x n ,t n ,h) is called the increment function. The increment function is 
selected to represent the average slope on the interval t e [t n , t n+ \\. In particular, 
the increment function has the form 

m 
a(x n ,t n ,h) = y^Cjkj, (10.66) 
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where m is called the order of the Runge-Kutta method, Cj are constants, and 
coefficients kj are obtained recursively according to 

h = f(x„,t n ) 

h — f(x n +ai,\hk\, t„ + p 2 h) 

h — f(x„ + a^ihki + a3 t2 hk 2 , t„ + pih) 



(m-i \ 

x„ + ^2 a m ,jhkj, t„ + p m h . (10.67) 

A more compact description of the Runge-Kutta methods is 

m 

x n +\ = x n + h y^ Cjkj, (10.68a) 

./ = ! 

where 

kj = / I x n + h ^ cijjki, t n + pjh | . (10.68b) 

To specify a particular method requires selecting a variety of coefficients (c;, fl/j, 
etc.). How is this to be done? 

We illustrate with examples. Suppose that m — 1. In this case 

x„+i = x„ + hc\k\ — x n + hc\f{x n , t„) (10.69) 

which gives (10.22) when c\ = 1. Thus, we are justified in calling the explicit 
Euler method a first-order Runge-Kutta method. 
Suppose that m — 2. In this case 

x n +l =x„ +hcif(x n ,t n ) + hc 2 f(x n + a 2 ,\hf (x n ,t n ),t n + p 2 h). (10.70) 

We observe that if we choose 

c 2 — c\ — 5, a 2 ,i = 1, P2 = 1, (10.71) 

then (10.70) reduces to 

x n +\ = x n + \h[f{x n , t n ) + f(x n + hf(x n , t n ), t n + h)], 

which is Heun's method [compare this with (10.47)]. Thus, we are justified in call- 
ing Heun's method a second-order Runge-Kutta method. However, the coefficient 
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choices in (10.71) are not unique. Other choices will lead to other second-order 
Runge-Kutta methods. We may arrive at a systematic approach for creating alter- 
natives as follows. 

For convenience as in (10.21), define x„ = x^\t n ). Since m — 2, we will 
consider the Taylor expansion 

x„+i =x n + hx n l) + ±h 2 xf } + 0(h 3 ) (10.72) 

[recall (10.52)] for which the term 0(h 3 ) simply denotes the higher-order terms. 
We recall that x^(t) — fix, t), so x„ — f(x n , t n ), and via the chain rule 

n \ df df dx df df 

x (2) (t) = —+——=—+— f(x, t), (10.73) 

dt dx at dt dx 

so (10.72) may be rewritten as 

x n +i =x„+ hfix n , t n ) + -h h -h f(x„, t„) + OQl ). 

2 dt 2 dx 

(10.74) 

Once again, the Runge-Kutta method for m — 2 is 

x n+ \ =x„ +hc\fix n ,t n ) + hc 2 fix n +a 2 ,ihki,t„ + p 2 h). (10.75) 

Recalling (10.26), the Taylor expansion of fix n + a 2 \hk\, t„ + p 2 h) is given by 

dfix n ,t n ) 



fix n + a 2 ,iMi, t„ + p 2 h) = fix n , t n ) + a 2z ihfix n , t n )- 



dx 



dfiX„, t„) r> 

p 2 h J \"' +Oih 2 ). (10.76) 

at 



Now we substitute (10.76) into (10.75) to obtain 

, 2 dfix n ,t n ) 



X n + \ = X„ + (Cl + C 2 )hfix„, t n ) + p 2 C 2 h z 



dt 



idfix„,t n ) i 

+ a 2A c 2 h 2 J \" fix n ,t n )+Oih 3 ). (10.77) 

dx 

We may now compare like terms of (10.77) with those in (10.74) to conclude that 
the coefficients we seek satisfy the nonlinear system of equations 

c\+c 2 — 1, p 2 c 2 — \, a 2t \c 2 — \. (10.78) 

To generate second-order Runge-Kutta methods, we are at liberty to choose the 
coefficients c\,c 2 , p 2 , and 02,1 in any way we wish so long as the choice sat- 
isfies (10.78). Clearly, Heun's method is only one choice among many possible 
choices. We observe from (10.78) that we have four unknowns, but possess three 
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equations. Thus, we may select one parameter "arbitrarily" that will then determine 
the remaining ones. For example, we may select c 2 , so then, from (10.78) 

1 1 

c\ = 1 - c 2 , p 2 = — , and a 2 ,i = — . (10.79) 

2c 2 2c 2 

Since one parameter is freely chosen, thus constraining all the rest, we say that 
second-order Runge-Kutta methods possess one degree of freedom. 

It should be clear that the previous procedure may be extended to systematically 
generate Runge-Kutta methods of higher order (i.e., m > 2). However, this requires 
a more complete description of the Taylor expansion for a function in two variables. 
This is stated as follows. 

Theorem 10.3: Taylor's Theorem Suppose that f(x,t), and all of its partial 
derivatives of order n + 1 or less are defined and continuous on D — {(x, t)\a < 
t <b,c < x < d}. Let (xq, to) e D; then, for all (x, t) e D, there is a point (rj, f ) € 
D such that 



r \ u t ^r-k, „ ^3 r /(^0,?0) 



f(^t) = J2\^J2( r k )o-toY- k (x-xo) 

A I ' I. n N / 



k=Q 

1 \— v / n + 1 







{n+\)\*-~i\ k l y dt n+i-k dx k 

k=0 

for which (17, f ) is on the line segment that joins the points (xo, to), and (x, t). 
Proof Omitted. 

The reader can now easily imagine that any attempt to apply this approach for 
m > 2 will be quite tedious. Thus, we shall not do this here. We will restrict our- 
selves to stating a few facts. Applying the method to m — 3 (i.e., the generation 
of third-order Runge-Kutta methods) leads to algorithm coefficients satisfying six 
equations with eight unknowns. There will be 2 degrees of freedom as a conse- 
quence. 

Fourth-order Runge-Kutta methods (i.e., m — 4) also possess two degrees of 
freedom, and also have a truncation error per step of 0(h 5 ). One such method 
(attributed to Runge) in common use is 

h 
x n+ i = x„ + - [hi + 21c 2 + 2k 3 + k 4 ] , (10.80) 

6 

where 

h = f(x n ,t n ) 

ki — f lx„ + \hk\ , t n + \h\ 

h — f \x„ + \hk 2 , t n + \h\ 

k 4 = f( Xn +hk 3 ,t n +h). (10.81) 

Of course, an infinite number of other fourth-order methods are possible. 
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We mention that Runge-Kutta methods are explicit methods, and so in principle 
carry some risk of instability. However, it turns out that the higher the order of 
the method, the lower the risk of stability problems. In particular, users of fourth- 
order methods typically experience few stability problems in practice. In fact, it 
can be shown that on applying (10.80) and (10.81) to the model problem (10.28), 
we obtain 



\+hX 



1 



h 2 X 2 



1 



h 3 X 3 



1 

24' 



h*X 4 



XQ. 



A plot of 



a = 1 + hX + -h 2 X 2 + -h 3 X 3 + —h 4 X A 
2 6 24 



(10.82) 



(10.83) 



in terms of hX appears in Fig. 10.8. To avoid linin^oo \x n \ = oo, we must have 
| cr | < 1, and so it turns out that 

-2.785 < hX < 

(see Table 9.11 on p. 685 of Rao [7]). This is in agreement with Fig. 10.8. Thus 

2.785 



X < 0, 



h < 



\x\ 



(10.84) 



This represents an improvement over (10.33). 

Example 10.8 Once again we repeat Example 10.5 for which 



dx 
dt 



— t 



2x 
t 



with xq — 1.0 for fo = -05, but here instead our step size is now h — 0.05. Addition- 
ally, our comparison is between Heun's method and the fourth-order Runge-Kutta 
method defined by Eqs. (10.80) and (10.81). 



b 1 




Figure 10.8 A plot of a in terms of hX [see Eq. (10.83)]. 
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Figure 10.9 Comparison of Heun's method (a second-order Runge-Kutta method) with 
a fourth-order Runge-Kutta method [Eqs. (10.80) and (10.81)]. This is for the differential 
equation in Example 10.8. 



The simulated solutions based on these methods appear in Fig. 10.9. As expected, 
the fourth-order method is much more accurate than Heun's method. The plot 
in Fig. 10.9 was generated using MATLAB, and the code for this appears in 
Appendix 10.A as an example. (Of course, previous plots were also produced by 
MATLAB codes similar to that in Appendix 10. A.) 



10.3 SYSTEMS OF FIRST-ORDER ODEs 

The methods of Section 10.2 may be extended to handle systems of first-order 
ODEs where the number of ODEs in the system is arbitrary (but finite). However, 
we will consider only systems of two first-order ODEs here. Specifically, we wish 
to solve (numerically) 

dX = fix, y, t), (10.85a) 



dt 
dy_ 

dt 



= g(x,y,t), 



(10.85b) 



where the initial condition is xq — x(to) and yo = y(to). This is sufficient, for 
example, to simulate the Duffing equation mentioned in Section 10.1. 



TLFeBOOK 



SYSTEMS OF FIRST-ORDER ODEs 443 

As in Section 10.2, we will begin with Euler methods. Therefore, following 
(10.21) we may Taylor-expand x(t) and y(t) about the sampling time t — t n . As 
before, for convenience we may define x„ — x^ k \t n ), y„ = y^ k \t n ). The relevant 
expansions are given by 

x n+1 = x„ + hx ( n l) + ±h 2 x ( n 2) + • • • , (10.86a) 

y n+1 =y n + hy n l) + \h 2 y^ + ■■■. (10.86b) 

The explicit Euler method follows by retaining the first two terms in each expansion 
in (10.86). Thus, the Euler method in this case is 

x n +\ — x„ + hf(x n , y„, t n ), (10.87a) 

y n +\ — y n + hg(x n , y n , t n ), (10.87b) 

where we have used the fact that x„ — f(x„,y„,t n ) and y n — g(x n , y„, t n ). As 
we might expect, the implicit form of the Euler method is 

x n +\ = x„ + hf(x n+ i, y n+ i, t n+ i), (10.88a) 

y n +\ — y n +hg(x n+ \,y n +\,t n +\). (10.88b) 

Of course, to employ (10.88) will generally involve solving a nonlinear system of 
equations for x n+ \ and y n +i, necessitating the use of Chapter 7 techniques. As 
before, we refer to parameter h as the step size. 

The accuracy of the explicit and implicit Euler methods for systems is the same 
as for individual equations; specifically, it is 0(h 2 ). However, stability analysis is 
more involved. Matrix methods simply cannot be avoided. This is demonstrated as 
follows. 

The model problem for a single first-order ODE was Eq. (10.28). For a coupled 
system of two first-order ODEs as in (10.85), the model problem is now 

dx 

— = ciqqx + ciQiy, (10.89a) 
dt 

dy 

— =a\ox + a\iy. (10.89b) 
dt 

Here a, ;j are real-valued constants. We remark that this may be written in more 
compact matrix form 

dx 

— = Ax, (10.90) 

dt 

where x — x(t) — [x(t)y(t)] and dx/dt = [dx(t)/dt dy(t)/dt] T and, of course 



«oo «oi 
fllO fl ll 



(10.91) 
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Powerful claims are possible using matrix methods. For example, it can be argued 
that if x(t) € R N (so that A € R NxN ), then dx(t)/dt — Ax(t) has solution 



At- 



x(t) = e m x(0) 



(10.92) 



for t > 0; that is, (10.92) for N = 2 is the general solution to (10.90) [and hence 
to (10.89)]. 5 

The constants in A are related to f(x, y, t) and g(x, y, t) in the following 
manner. Recall (10.25). If we retain only the linear terms in the Taylor expansions 
of / and g around the point (xq, yo, to), then 

,, ,. ,, , , . , df(xo,yo,to) df(xo,yo,t ) 

f(x, y, t) « f{x , y , t ) + (x - x ) h (y - yo)- 



dx 



+ (t- to) 



dfjxp, yp, t ) 
dt 



dg(xo, yo, to) 
g(x, y, t) «s g (x , y , t ) + (x - xq) h (y - yo) 



9v 

(10.93a) 
3^(^o- yo, to) 



i)x 



dy 



+ (t- to) 
As a consequence 



dgjxp, yp, t ) 
dt 



df(xo,yo,to) dfjxp, yp, tp) 

dx dy 

dg(x , yo, t ) dg(x Q , yo, to) 



(10.93b) 



dx 



dy 



(10.94) 



At this point we may apply the explicit Euler method (10.87) to the model problem 
(10.89), which results in 



x n +i 

Jn+l 



1 + haoo haoi 
ha\o 1 + han 



Xn 

y n 



(10.95) 



Yes, as surprising as it seems, although At is a matrix exp(Ar) makes sense as an operation. In fact, 
for example, with x(t) = [xQ(t) ■ ■ ■ x n _\(t)] , the system of first-order ODEs 

dx(t) 



dt 



Ax(t) + by(t) 



(A 6 R" x ", b 6 R", and y(t) e R) has the general solution 

x(t) = e At x(0~)+ e A(t ~ T) by(T)dT. 

Jo- 

The integral in this solution is an example of a convolution integral. 
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An alternative form for this is 

x n +i = V + hA)x n . (10.96) 

Here / is a 2 x 2 identity matrix and x n — [x n y n ] T . With xo = [xq yo] T , we 
may immediately claim that 

x„ = (I + hA) n x , (10.97) 

where n e Z + . We observe that this includes (10.32) as a special case. Naturally, 
we must select step size h to avoid instability; that is, we are forced to select h 
to prevent lim„^oo ||x„|| = oo. In principle, the choice of norm is arbitrary, but 
2-norms are often chosen. We recall that there is a nonsingular matrix T (matrix 
of eigenvectors) such that 

T~ l [I + hA]T = A, (10.98) 

where A is the matrix of eigenvalues. We will assume that 



A 



*o 
A.i 



(10.99) 



In other words, we assume / + hA is diagonalizable . This is not necessarily always 
the case, but is an acceptable assumption for present purposes. Since from (10.98) 
we have I + hA = TAT' 1 , (10.96) becomes 

x n+ i = TAT~ l x n , 



T~ l x n+l = AT~ l x n . (10.100) 
Let y n — T~ lr x n , so therefore (10.100) becomes 

y n+l =Ay n . (10.101) 

In any norm lim n ^oo ||7„|| ^ oo, provided |A.&| < 1 for all k = 0, 1, . . . , N — 1 



< 



(A € R ). Consequently, liirin^oo \\x n \\ ^ oo too (because ||3c„|| = ||7 , y„|| 
||r|| ||y„|| and ||r|| is finite). We conclude that h is an acceptable step size, 
provided the eigenvalues of / + hA do not possess a magnitude greater than unity. 
Note that in practice we normally insist that h result in |A.jt| < 1 for all k. 

We may apply the previous stability analysis to the implicit Euler method. 
Specifically, apply (10.88) to model problem (10.89), giving 

X n +l — X n + h[aooX n+ i + Ooijn+l] 
y n +l — y n + h[a w x n+ i + a n y n+l ], 
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which in matrix form becomes 





Xn+l 

. yn+i 


= 


Xn 

. yn 


+ h 


aoo fl oi 
«io an 




x n + \ 
_ 3'n+l 


or more compactly as 

Xn+l 


= X n -\- rlAXn-\-\. 


Consequently, for n e Z + 













{[l-hATTx . 



(10.102) 



(10.103) 



For convenience we can define B — [I — hA]~ l . Superficially, (10.103) seems to 
have the same form as (10.97). We might be lead therefore to believe (falsely) 
that the implicit method can be unstable, too. However, we may assume that there 
exists a nonsingular matrix V such that (if A e R NxN ) 



V~ l AV = r, 



(10.104) 



where V — diag(yo, y\, . . , , yw-i), which is the diagonal matrix of the eigenvalues 
of A. (Once again, it is not necessarily the case that A is always diagonalizable, 
but the assumption is reasonable for our present purposes.) Immediately 



[/ - hA]~ l =U- hVTV~ l r l 



(V[V~ l V -hT]V~ l ) 



K-l 



so that 

[/ - /jA]" 1 = V[I - hTT X V~ x . 

Consequently x n+ \ — [I — hA\~ x x n becomes 



x n+l = VU-hTT l V- x x n . 



(10.105) 



(10.106) 



Define y n — V x n , and so (10.106) becomes 

y„+i = [I -hTY x y n - 



(10.107) 



Because / — hT is a diagonal matrix, a typical main diagonal element of [/ — 
hT] -1 is ak — 1/(1 — hyk). It is a fact (which we will not prove here) that the 
model problem in the general case is stable provided the eigenvalues of A all pos- 
sess negative-valued real parts. 6 Thus, provided Re(y^) < for all k, we are assured 
that \ok | < 1 for all k, and hence lim„^oo | \~y n \ \ —0. Thus, lim n ^oo | |x„ 1 1 =0, too, 
and so we conclude that the implicit form of the Euler method is unconditionally 
stable. Thus, if the model problem is stable, we may select any h > 0. 

The eigenvalues of A may be complex-valued, and so it is the real parts of these that truly determine 
system stability. 
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The following example is of a linear system for which a mathematically exact 
solution can be found. [In fact, the solution is given by (10.92).] 

Example 10.9 Consider the ODE system 

— = -2x+±y, (10.108a) 
at 4 

— = -3x. (10.108b) 
at 

The initial condition is xq — x(0) = 1, yo — y(0) — — 1. From (10.94) we see that 

A = 



-2 i 

Z 4 



-3 

The eigenvalues of A are yo = — j, and y\ — — |. These eigenvalues are both 
negative, and so the solution to (10.108) happens to be stable. In fact, the exact 
solution can be shown to be 

x(t) = -|e" r/2 + je _3 ' /2 , (10.109a) 

y(t) = -|e" r/2 + |e- 3(/2 (10.109b) 

for f > 0. Note that the eigenvalues of A appear in the exponents of the exponentials 
in (10.109). This is not a coincidence. The explicit Euler method has the iterations 

x n +\ — x n + h \-2x n + \y n \ , (10.110a) 

y n +\ = y n ~ 3hx„. (10.110b) 

Simulation results are shown in Figs. 10.10 and 10.11 for h — 0.1 and h — 
1.4, respectively. This involves comparing (10.110a,b) with the exact solution 
(10.109a,b). Figure 10.10b shows a plot of the eigenvalues of / + hA for various 
step sizes. We see from this that choosing h = 1 .4 must result in an unstable simu- 
lation. This is confirmed by the result in Fig. 10.11. For comparison purposes, the 
eigenvalues of / + hA and of [/ — hA] -1 are plotted in Fig. 10.12. This shows 
that, at least in this particular case, the implicit method is more stable than the 
explicit method. 



Example 10.10 Recall the Duffing equation of Section 10.1. Also, recall the 
fact that this ODE can be rewritten in the form of (10.85a,b), and this was done in 
Eq. (10.4a,b). 

Figures 10.13 and 10.14 show the result of simulating the Duffing equation 
using the explicit Euler method for the model parameters 

F = 0.5, «=1, m=\, a = \, 5 = 0.1, k = 0.05. 
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Figure 10.10 Simulation results for h = 0.1. Plot (b) shows the eigenvalues of / + hA. 
Both plots were obtained by applying the explicit form of the Euler method to the ODE 
system of Example 10.9. Clearly, the simulation is stable for h — 0.1. 



Thus (10.1) is now 



d x 



dx 



—r = 0.5 cos(f)- 0.05 [x + O.lx*]. 

dt 2 dt 

We use initial condition x(0) = y(0) = 0. The driving function (applied force) 
0.5 cos(f ) is being opposed by the restoring force of the spring (terms in square 
brackets) and friction (first derivative term). Therefore, on physical grounds, we do 
not expect the solution x(t) to grow without bound as t — > oo. Thus, the simulated 
solution to this problem must be stable, too. 

We mention that an analytical solution to the differential equation that we are 
simulating is not presently known. 

From (10.94) for our Duffing system example we have 



A = 





a 35 



1 



-— x — 



k 
mm m 

Example 10.11 According to Hydon [1, p. 61], the second-order ODE 



d 2 x 



(dxV 1 

V dt ) x 



1\ dx 

x I dt 



(10.111) 
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Figure 10.11 This is the result of applying the explicit form of the Euler method to the 
ODE system of Example 10.9. Clearly, the simulation is not stable for h = 1.4. This is 
predicted by the eigenvalue plot in Fig. 10.10b, which shows that one of the eigenvalues of 
/ + hA has a magnitude exceeding unity for this choice of h. 
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Figure 10.12 Plots of the eigenvalues of / + hA (a), which determine the stability of the 
explicit Euler method and the eigenvalues of [/ — hA]~ (b), which determine the stability 
of the implicit Euler method. This applies for the ODE system of Example 10.9. 
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Figure 10.13 (a) Explicit Euler method simulation of the Duffing equation; (b) magnitude 
of the eigenvalues of I + hA. Both plots were obtained by applying the explicit form of the 
Euler method to the ODE system of Example 10.10, which is the Duffing equation expressed 
as a coupled system of first-order ODEs. The simulation is apparently stable for h = 0.02. 
This is in agreement with the prediction based on the eigenvalues of I + hA [plot (b)], 
which have a magnitude of less than unity for this choice of h. 



has the exact solution 



x(t) 



c\ 



C\ 



- 1 tanh(Jc( - \{t + c 2 )), 
ci -it + c 2 )-\ 



1 -cf tanh(./l -cf(f + c 2 )), 



c\ > 1 



c\ < 1 



(10.112) 



The ODE in (10.111) can be rewritten as the system of first-order ODEs 



dx 




— 


— y> 


dt 




dy 


t 


dt 


X 



(10.113a) 
(10.113b) 
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Figure 10.14 (a) Explicit Euler method simulation of the Duffing equation; (b) magnitude 
of the eigenvalues of / + hA. Both plots were obtained by applying the explicit form of the 
Euler method to the ODE system of Example 10.10, which is the Duffing equation expressed 
as a coupled system of first-order ODEs. The simulation is not stable for h = 0.055. This is 
in agreement with the prediction based on the eigenvalues of I + hA [plot (b)], which have 
a magnitude exceeding unity for this choice of h. 

The initial condition is xq — x(0), and y(0) = -^p-\ t= Q. From (10.94) we have 

1 



A = 



X Q \ X Q / 



2yo ^ ( 1 

xq V XQ 



(10.114) 



Using (10.113a), we may obtain y(t) from (10.112). For example, let us consider 

„2 



simulating the case c\=\. Thus, in this case 



y(t) 



1 



(f + C 2 ) 2 ' 



(10.115) 



For the choice c\ — 1, we have 



xq — c\ , 

C2 



yo = -T- 



(10.116a) 
(10.116b) 
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Figure 10.15 Magnitude of the eigenvalues of / + hA is shown in plot (b). Both plots 
show the simulation results of applying the explicit Euler method to the ODE system in 
Example 10.11. The simulation is (as expected) stable for h = 0.02. Clearly, the simulated 
result agrees well with the exact solution. 



If we select c\ = 1, then 



xq — 1 , yo — (1 - x ) . 



(10.117) 



Let us assume xq — — 1, and so yo — 4. The result of applying the explicit Euler 
method to system (10.113) with these conditions is shown in Figs. 10.15 and 10.16. 
The magnitudes of the eigenvalues of / + hA are displayed in plots (b) of both 
figures. We see that for Fig. 10.15, h — 0.02 and a stable simulation is the result, 
while for Fig. 10.16, we have h — 0.35, for which the simulation is unstable. This 
certainly agrees with the stability predictions based on finding the eigenvalues of 
matrix I + hA. 

Examples 10.10 and 10.11 illustrate just how easy it is to arrive at differential 
equations that are not so simple to simulate in a stable manner with low-order 
explicit methods. It is possible to select an h that is "small" in some sense, yet 
not small enough for stability. The cubic nonlinearity in the Duffing model makes 
the implementation of the implicit form of Euler' s method in this problem quite 
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Figure 10.16 Plot (b) shows magnitude of eigenvalues of / + hA. Both plots show the sim- 
ulation results of applying the explicit Euler method to the ODE system in Example 10.11. 
The simulation is (as expected) not stable for h — 0.35. Instability is confirmed by the fact 
that the simulated result deviates greatly from the exact solution when t is sufficiently large. 



unattractive. So, a better approach to simulating the Duffing equation is with a 
higher-order explicit method. 

For example, Heun's method for (10.85) may be stated as 



x„+i = x„ + jh[f(x n ,y n ,t„) + f(x n +ki,y„ + h,t„ + h)], 



y n +\ — y n + 2 h \-8(Xn,yn,t n ) + g(Xn +kl,y„ + h,t„ + h)], 



where 



k\ = hf(x„, y„, t n ), h = hg(x„, y„, t„). 



(10.118a) 
(10.118b) 

(10.119) 



Also, for example, Chapter 36 of Branson [8] contains a summary of higher-order 
methods that may be applied to (10.85). 

Example 10.12 Recall the Duffing equation simulation in Example 10.10. 
Figure 10.17 illustrates the simulation of the Duffing equation using both the 
explicit Euler method and Heun's method for a small h (i.e., h — 0.005 in both 

cases). 
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Figure 10.17 Comparison of the explicit Euler (a) and Heun (b) method simulations of 
the ODE in Example 10.12, which is the Duffing equation. Here the step size h is small 
enough that the two methods give similar results. 



At this point we note that there are other ways to display the results of numerical 
solutions to ODEs that can lead to further insights into the behavior of the dynamic 
system that is modeled by those ODEs. Figure 10.18 illustrates the phase portrait 
of the Duffing system. This is obtained by plotting the points (x n ,y n ) on the 
Cartesian plane, yielding an approximate plot of (x(t),y(t)). The resulting curve 
is the trajectory, or orbit for the system. Periodicity of the system's response is 
indicated by curves that encircle a point of equilibrium, which in this case would 
be the center of the Cartesian plane [i.e., point (0, 0)]. The trajectory is tending to 
an approximately ellipse-shaped closed curve indicative of approximately simple 
harmonic motion. 

The results in Fig. 10.18 are based on the parameters given in Example 10.10. 
However, Fig. 10.19 shows what happens when the system parameters become 



F = 0.3, (o=l, m=\, a = -\, 5=1, k = 0.22. 



(10.120) 



The phase portrait displays a more complicated periodicity than what appears in 
Fig. 10.18. The figure is similar to Fig. 2.2.5 in Guckenheimer and Holmes [11]. 
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Figure 10.18 (a) The result of applying Heun's method to obtain the numerical solution 
of the Duffing system specified in Example 10.10; (b) the phase portrait for the system 
obtained by plotting the points (x n , y n ) [from plot (a)] on the Cartesian plane, thus yielding 
an approximate plot of (x(t), y(t)). 



As in the cases of the explicit and implicit Euler methods, we may obtain a 
theory of stability for Heun's method. As before, the approach is to apply the 
model problem (10.90) to (10.118) and (10.119). As an exercise, the reader should 
show that this yields 



X n + l 



hA 



\h\ 



%n-> 



(10.121) 



where / is the 2x2 identity matrix, A is obtained by using (10.94), and, of course, 
I„ = [x n y n ] T . [The similarity between (10.121) and (10.48) is no coincidence.] 
Criteria for the selection of step size h leading to a stable simulation can be 
obtained by analysis of (10.121). But the details are not considered here. 



10.4 MULTISTEP METHODS FOR ODEs 

The numerical ODE solvers we have considered so far were either implicit methods 
or explicit methods. But in all cases they were examples of so-called single-step 
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Figure 10.19 (a) The result of applying Heun's method to obtain the numerical solu- 
tion of the Duffing system specified in Example 10.12 [i.e., using the parameter values in 
Eq. (10.120)]; (b) the phase portrait for the system obtained by plotting the points (x„, y n ) 
[from plot (a)] on the Cartesian plane, thus yielding an approximate plot of (x(t), y(f))- 



methods; that is, x n +\ was ultimately only a function of x n . A disadvantage of 
single-step methods is that to achieve good accuracy often requires the use of 
higher-order methods (e.g., fourth- or fifth-order Runge-Kutta). But higher-order 
methods need many function evaluations per step, and so are computationally 
expensive. 

Implicit single-step methods, although inherently stable, are not more accurate 
than explicit methods, although they can track fast changes in the solution x(t) 
better than can explicit methods (recall Example 10.5). However, implicit methods 
may require nonlinear system solvers (i.e., Chapter 7 methods) as part of their 
implementation. This is a complication that is also not necessarily very efficient 
computationally. Furthermore, the methods in Chapter 7 possess their own stability 
problems. Therefore, in this section we introduce multistep predictor-corrector 
methods that overcome some of the deficiencies of the methods we have considered 
so far. 

In this section we return to consideration of a single first-order ODE IVP 



„i dx(t) 

c (D (f) = _AZ = f( x(f)t t)t Xo = x(0) . 
dt 



(10.122) 
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In reality, we have already seen a single-step predictor-corrector method in 
Section 10.2. Suppose that we have the following method: 

x n +\ — x n + hf(x n , t n ) (predictor step) (10.123a) 

x„+\ — x n + \h[f{x n +\Jn+i) + f(x„, t„)] (corrector step). (10.123b) 

If we substitute (10.123a) into (10.123b), we again arrive at Heun's method 
[Eq. (10.47)], which overcame the necessity to solve for x n +i in the implicit method 
of Eq. (10.59) (trapezoidal method). Generally, predictor-corrector methods replace 
implicit methods in this manner, and we will see more examples further on in 
this section. When a higher-order implicit method is "converted" to a predictor- 
corrector method, the need to solve nonlinear equations is eliminated and accuracy 
is preserved, but the stability characteristics of the implicit method will be lost, at 
least to some extent. Of course, a suitable stability theory will still allow the user 
to select reasonable values for the step size parameter h. 

We may now consider a few simple examples of multistep methods. Perhaps 
the simplest multistep methods derive from the numerical differentiation ideas from 
Section 9.6 (of Chapter 9). Recall (9.138), for which 

x m {t) = —[x(t + h)- x(t - h)] - -h 2 x (3 \i;) (10.124) 

2h 6 

for some f € [t — h, t + h]. The explicit Euler method (10.22) can be replaced 
with the midpoint method derived from [using t — t n in (10.124)] 

1 

f(x(t n ), t n ) RS —[ x (t n+ i) - X(t n -l)], 

2h 
so this method is 

X n +\ = X n -\ + 2hf(x„, t n ). (10.125) 

This is an explicit method, but x n +\ depends on x„-\ as well as x n . We may call 
it a two-step method. Similarly, via (9.153) 

1 
f(x(t n ), t n ) % —l-3x(t n ) + Ax(t n+ i) - x(t n+2 )], 
2h 

so we have the method 

x n +2 = 4x„+i - 3x n - 2hf(x n , t n ) 
which can be rewritten as 

x n+ \ = 4x n - 3x„_i - 2hf(x n -i, t„-\). (10.126) 
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Finally, via (9.154), we obtain 

1 

f(x(t„), t„) « — -[x(t n -2) - Ax{t n -\) + 3x(f„)], 

2« 
yielding the method 

x n+ i = \x n - \x n -\ + \hf(x n+ \, t n +i). (10.127) 

Method (10.126) is a two-step method that is explicit, but (10.127) is a two-step 
implicit method since we need to solve for x n+ \. 

A problem with multistep methods is that the IVP (10.122) provides only one 
initial condition x . But for n — 1 in any of (10.125), (10.126), or (10.127), we 
need to know xi\ that is, for two-step methods we need two initial conditions, or 
starting values. A simple way out of this dilemma is to use single-step methods to 
provide any missing starting values. In fact, predictor-corrector methods derived 
from single-step concepts (e.g., Runge-Kutta methods) are often used to provide 
the starting values for multistep methods. 

What about multistep method accuracy? Let us consider the midpoint method 
again. From (10.124) for some % n e [f„_i, t n+ {\ 

X(fn+l) = x{t n -i) + 2hf(x(t n ), t n ) + \hW>{t- n ), (10.128) 

so the method (10.125) has a truncation error per step that is 0(/z 3 ). The midpoint 
method is therefore more accurate than the explicit Euler method [recall (10.43) 
and (10.44)]. Yet we see that both methods need only one function evaluation 
per step. Thus, the midpoint method is more efficient than the Euler method. We 
recall that Heun's method (a Runge-Kutta method) has a truncation error per 
step that is 0(h?), too [recall (10.56)], and so Heun's method may be used to 
initialize (i.e., provide starting values for) the midpoint method (10.125). This 
specific situation holds up in general. Thus, a multistep method can often achieve 
comparable accuracy to single-step methods, and yet use fewer function calls, 
leading to reduced computational effort. 

What about stability considerations? Let us continue with our midpoint method 
example. If we apply the model problem (10.28) to (10.125), we have the difference 
equation 

x n+ \ = x„_i + 2hXx n , (10.129) 

which is a second-order difference equation. This has characteristic equation 7 

z 2 - 2hXz - 1 = 0. (10.130) 

We may rewrite (10.129) as 



which has the z-transform 



x n+2 -2hXx n+ \ -x n 



(z 2 -2h\z-l)X(z) = 0. 
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For convenience, let p — hk, in which case the roots of (10.130) are easily seen 
to be 



zi = p + yjp 2 + l, Z2 = /0-VP 2 + '•• (10.131) 

A general solution to (10.129) will have the form 

Xn = c x z\ + c 2 z\ (10.132) 

for n e Z + . Knowledge of xo and x\ allows us to solve for the constants c\ and c 2 in 
(10.132), if this is desired. However, more importantly, we recall that we assume 
k < 0, and we seek step size ft > so that lim n ^oo \x n \ ^ oo. But in this case 
p = hk <0, and hence from (10.131), \z 2 \ > I for all h > 0. If c 2 #0 in (10.132) 
(which is practically always the case), then we will have linin^oo \x„ \ — oo ! Thus, 
the midpoint method is inherently unstable under all realistic conditions ! Term c\z\ 
in (10.132) is "harmless" since |zi| < 1 for suitable h. But the term c 2 z 2 , often 
called a parasitic term, will eventually "blow up" with increasing n, thus fatally 
corrupting the approximation to x(t). 

Unfortunately, parasitic terms are inherent in multistep methods. However, there 
are more advanced methods with stability theories designed to minimize the effects 
of the parasitics. We now consider a few of these improved multistep methods. 

10.4.1 Adams-Bashforth Methods 

Here we look at the Adams-Bashforth (AB) family of multistep ODE IVP solvers. 
Section 10.4.2 will look at the Adams-Moulton (AM) family. Our approach follows 
Epperson [12, Section 6.6]. Both families are derived using Lagrange interpolation 
[recall Section 6.2 from Chapter 6 (above)]. 

Recall (10.122) which we may integrate to obtain (t„ = to + nh) 

X(fn+l) = X{t n ) + f" ¥1 f(x(t),t)dt. (10.133) 

Now suppose that we had the samples x(t n -k) for k — 0, 1, . . . , m (i.e., m + 1 sam- 
ples of the exact solution x{t)). Via Lagrange interpolation theory, we may interpo- 
late F(t) — f(x(t), t) [the integrand of (10.133)] for t e [t„- m , £ n +i] 8 according to 

in 

Pm(t) = J^L k (t)f(x(t n - k ),t n - k ), (10.134) 

k=Q 

A solution to (10.129) exists only if z — 2hkz — 1 = 0. If the reader has not had a signals and systems 
course (or equivalent) then this reasoning must be accepted "on faith." But it may help to observe that 
the reasoning is similar to the theory of solution for linear ODEs in constant coefficients. 

The upper limit on the interval [t n — m , t„] has been extended from t n to t n+ \ here. This is allowed 
under interpolation theory, and actually poses no great problem in either method development or error 
analysis. We are using the Lagrange interpolant to extrapolate from t = t n to t n j r \. 
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where 

m 

£*(*) = [I t ~ t l~ l ■ (10.135) 



i-0 

ijtk 



From (6.14) for some % t e [t n - m , t n+ \] 

1 m 

F(t) = Pm (t) + ———F (m+l) ^ t )V[(t - tn-i). 

(m + 1)! jj^ 

However, F(t) — f(x(t), t) — x m (t) so (10.136) becomes 

.. m 

F{t) = p m (t) + . X {m+2) %) Y\(f - tn-i). 

(m + 1)! ll 

1=0 

Thus, if we now substitute (10.137) into (10.133), we obtain 

dn+l 

f(x(t„-k), t n -k) I 

A — -- i J 

where 

— X v "^>(^t) 

nt 

(=0 



(10.136) 



(10.137) 



c(t n+l )=x(t n ) + 2_^f(x(t n - k ),t n - k ) L k (t)dt + R m (t„+i), (10.138) 



*mfe+l) = / — X {m+2 \Ht) Y\(f - tn-i) dt. (10.139) 

Jt„ (m + 1)! H; 



Polynomial jt(t) does not change sign for t e [t n ,t n +i] (which is the interval of 
integration). Thus, we can say that there is a % n e [t n , t n +i] such that 



f'n+l 

Jt„ 



R m (t n+ i) = - ] — x {m+2 \Hn) I n(i)dt. (10.140) 

(m + 1)! 



For convenience, define 
1 f'"+> 



1 ftn+l 1 [tn-l 

(m + 1)! J, (m + 1)!./. 



(t - t n )(t - t„-\) ■■■(t - t n -m) dt 

(10.141) 
and 

f'n+l 

k k = / L k (t)dt. (10.142) 

Thus, (10.138) reduces to [with R m (t n+ i) = p m x (m+2) (f„)] 

m 

x(t n+l ) = *(*„) + £ X k f(x(t n - k ), t n - k ) + A„* (m+2) (? B ) (10.143) 

>t=o 
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TABLE 10.1 Adams-Bashforth Method Parameters 



m 


A 


X l 


X 2 


*3 


^m(fn+l) 





h 








^ 2 * (2 >(£„) 


1 


Ik 

2 


1 
— h 

2 






^V 3 >(?„) 


2 


23 

—h 

12 


12 


Ik 

12 




Vx (4) &) 

O 


3 


55 
— h 
24 


59 

h 

24 


37 

—h 

24 


9 
h 

24 


251 5 r<n 
720 



for some §„ € [?„, f„+i]. The order m + 1 Adams-Bashforth method is therefore 
defined to be 

m 
*n+l = *n + ^W(*n-*> '«-*)■ (10.144) 

It is an explicit method involving m + 1 steps. Table 10.1 summarizes the method 
parameters for various m, and is essentially Table 6.6 from Ref. 12. 



10.4.2 Adams-Moulton Methods 

The Adams-Moulton methods are a modification of the Adams-Bashforth meth- 
ods. The Adams-Bashforth methods interpolate using the nodes t n , t„-i, . . . , t n - m . 
On the other hand, the Adams-Moulton methods interpolate using the nodes 
t n +i, t n , ■ ■ ■ , tn-m+i- Note that the number of nodes is the same in both methods. 
Consequently, (10.138) becomes 

x(t„+i) = x{t n ) + 2^ f(x(tn-k),tn-k) / L k (t)dt + R m (t n+ i), (10.145) 

k=-\ Jt " 



where now 

m — 1 



I tn 



Lk (t) = n -^-^ 

, tn— k *n 



(10.146) 






and ff m (*„ + i) = p m x (m+2) (t; n ) with 



1 /""+ 



Pm = 7 — TTT / ( f -?»+l)('-'«)- ••(^-fn-m+l) <*f. (10.147) 

(m + ' 
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Thus, the order m + 1 Adams-Moulton method is defined to be 

m— 1 



x n +i = x„ + ^2 X k f(x n -k, t„- k ), (10.148) 



where 

X k = I ¥1 L k (t)dt (10.149) 



/ 

Jt„ 



[same as (10.142) except k — -1, 0, 1, . . . , m - 1, and L k {t) is now (10.146)]. 
Method (10.148) is an implicit method since it is necessary to solve for x n+ \. 
It also requires m + 1 steps. Table 10.2 summarizes the method parameters for 
various m, and is essentially Table 6.7 from Ref. 12. 

10.4.3 Comments on the Adams Families 

For small values of m in Tables 10.1 and 10.2, we see that the Adams families 
(AB family and AM family) correspond to methods seen earlier. To be specific: 

1. For m = in Table 10.1, Xq — h, so (10.144) yields the explicit Euler method 
(10.22). 

2. For m — in Table 10.2, X-i — h, so (10.148) yields the implicit Euler 
method (10.35). 

3. For m = 1 in Table 10.2, A_i = Xq — jh, so (10.148) yields the trapezoidal 
method (10.59). 

Stability analysis for members of the Adams families is performed in the usual 
manner. For example, when m = 1 in Table 10.1 (i.e., consider the second-order 
AB method), Eq. (10.144) becomes 

x n +\ — x„ + jh[3f(x n , t n ) - f(x n -\,t n -\)]. (10.150) 



TABLE 10.2 Adams-Moulton Method Parameters 



in 


*-l 


*0 


*1 


^2 


Rm(t n +\) 





h 








-Vx (2 %„) 


1 


u 


1 
-h 






- — h 3 x 0) tf„) 




2 


2 






12 


? 


5 
— h 


A. 


1 

h 




_1^(4) (?) 




12 


12 


12 




24 




9 


19 


5 


1 


_ii fc 5 x (5) ft) 


-! 


— h 


— h 


h 


— h 




24 


24 


24 


24 


720 
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Application of the model problem (10.28) to (10.150) yields 
x„+i = f 1 + -hX\x n - -h\x n _i 



or 



Xn+2 ~ ( 1 + -hk j X n +1 + -hXx n = 0. 



This has characteristic equation (with p — hX) 



-r - ! 1 + 2 P ) Z+ 2 P ^°' 



(10.151) 



This equation has roots 



z\ 



Z2 



l+-p)+^l+p + -p 2 



1+-P)-^1+P + -P 2 



(10.152) 



We need to know what range of h > yields \z 1 1 , | zi \ < 1 . We consider only p < 
since X < 0. Figure 10.20 plots |zi| and \z2\ versus p, and suggests that we may 
select h such that 

-1<AA,<0. (10.153) 

Plots of stability regions for the other Adams families members may be seen in 
Figs. 6.7 and 6.8 of Epperson [12]. Note that stability regions occupy the complex 
plane as it is assumed in such a context that X e C. However, we have restricted 
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2 
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1 
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i i i i i i i i T ** — - 
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-0.8 -0.6 -0.4 -0.2 



Figure 10.20 Magnitudes of the roots in (10.152) as a function of p = hX. 
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our attention to a first-order ODE IVP here, and so it is actually enough to assume 
that X e R. 

Finally, observe that AB and AM methods can be combined to yield predictor- 
correctors. An mth-order AB method can act as a predictor for an mth-order 
AM method that is the corrector. A Runge-Kutta method can initialize the 
procedure. 



10.5 VARIABLE-STEP-SIZE (ADAPTIVE) METHODS FOR ODEs 

Accuracy in the numerical solution of ODEs requires either increasing the order 
of the method applied to the problem or decreasing the step-size parameter h. 
However, high-order methods (e.g., Runge-Kutta methods of order exceeding 5) 
are not very attractive at least in part because of the computational effort involved. 
To preserve accuracy while reducing computational requirements suggests that we 
should adaptively vary step size h. 

Recall Example 10.5, where we saw that low order methods were not accurate 
near t — to- For a method of a given order, we would like in Example 10.5 to 
have a small h for t near to, but a larger h for t away from to- This would reduce 
the overall number of function evaluations needed to estimate x(t) for t e [to, f/]. 
The idea of adaptively varying h from step to step requires monitoring the error 
in the solution somehow; that is, ideally, we need to infer e n — \x(t n ) — x n \ [x„ 
is the estimate of x(t) at t — t n from some method] at step n. If e n is small 
enough, h may be increased in size at the next step, but if e n is too big, we 
decrease h. 

Of course, we do not know x(t n ), so we do not have direct access to the error e n . 
However, one idea that is implemented in modern software tools (e.g., MATLAB 
routines ode23 and ode45) is to compute x n for a given h using two methods, 
each of a different order. The method of higher order is of greater accuracy, so 
if x n does not differ much between the methods, we are lead to believe that h is 
small enough, and so may be increased in the next step. On the other hand, if the 
x n values given by the different methods significantly vary, we are then lead to 
believe that h is too big, and so should be reduced. 

In this section we give only a basic outline of the main ideas of this process. 
Our emphasis is on the Runge-Kutta-Fehlberg (RKF) methods, of which MAT- 
LAB routines ode23 and ode45 are particular implementations. Routine ode23 
implements second- and third-order Runge-Kutta methods, while ode45 imple- 
ments fourth- and fifth-order Runge-Kutta methods. Computational efficiency is 
maintained by sharing intermediate results that are common to both second- and 
third-order methods and common to both fourth- and fifth-order methods. More 
specifically, Runge-Kutta methods of consecutive orders have constants such as kj 
[recall (10.67)] in common with each other and so need not be computed twice. 
We mention that ode45 implements a method based on Dormand and Prince [14], 
and a more detailed account of this appears in Epperson [12]. An analysis of the 
RKF methods also appears in Burden and Faires [17]. The details of all of this are 
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quite tedious, and so are not presented here. It is also worth noting that an account 
of MATLAB ODE solvers is given by Shampine and Reichelt [13], who present 
some improvements to the older MATLAB codes that make them better at solving 
stiff systems (next section). 

A pseudocode for something like ode45 is as follows, and is based on Algo- 
rithm 6.5 in Epperson [12]: 

Input tQ,XQ\ { initial condition and starting time } 

Input tolerance e > 0; 

Input the initial step size h > 0, and final time tf > frj; 

n:=0; 

while t n < tf do begin 

X-| := RKF4(x n , t n ,h); { 4th order RKF estimate of x n+1 } 

X 2 :=RKF5(x n ,t n ,h)\ 

E:=|X 1 -X 2 |; 

if Xhe < E < hi then begin { h is OK J 

x n+1 :=*2; 

f„+1 :=t n +h; 

n :=n+ 1; 
else if E > he then { h is too big } 

h := h/2; { reduce h and repeat } 
else { h is too small } 

h :=2h; 

x n+\ -=X2\ 

f„+1 :=t n +h; 

n :=n + 1; 
end; 
end; 



Of course, variations on the "theme" expressed in this pseudocode are possible. As 
noted in Epperson [12], a drawback of this algorithm is that it will tend to oscillate 
between small and large step size values. We emphasize that the method is based on 
considering the local error in going from time step t n to t n +\. However, this does 
not in itself guarantee that the global error \x{t n ) — x n \ is small. It turns out that if 
adequate smoothness prevails [e.g., if fix, t) is Lipschitz as per Definition 10.1], 
then small local errors do imply small global errors (see Theorem 6.6 or 6.7 in 
Ref. 12). 

Example 10.13 This example illustrates a typical application of MATLAB 
routine ode23 to the problem of simulating the Colpitts oscillator circuit of Example 
10.2. 

Figure 10.21 shows a typical plot of i>cE(t), and the phase portrait for parameter 
values 

V TH = 0.75 V (volts), V CC = 5 (V), V EE = -5 (V), R EE = 400 Q, (ohms), 
R L = 35 (Q), L = 98.5 x 10" 6 H (henries), p F = 200, R n = 100 (f2), 
d = C 2 = 54 x 10" 9 F (farads). (10.154) 
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Figure 10.21 Chaotic regime (a) phase portrait of Colpitts oscillator and (b) collector- 
emitter voltage, showing typical results of applying MATLAB routine ode23 to the simula- 
tion of the Colpitts oscillator circuit. Equation (10.154) specifies the circuit parameters for 
the results shown here. 



These circuit parameters were used in Kennedy [9], and the phase portrait in 
Fig. 10.21 is essentially that in Fig. 5 of that article [9]. The MATLAB code that 
generates Fig. 10.21 appears in Appendix 10. B. 

For the parameters in (10.154) the circuit simulation phase portrait in Fig. 10.21 
is that of a strange attractor [11, 16], and so is strongly indicative (although not 
conclusive) of chaotic dynamics in the circuit. 

We note that under "normal" circumstances the Colpitts oscillator is intended to 
generate sinusoidal waveforms, and so the chaotic regime traditionally represents 
a failure mode, or abnormal operating condition for the circuit. However, Kennedy 
[9] suggests that the chaotic mode of operation may be useful in chaos-based data 
communications (e.g., chaotic-carrier communications). 

The following circuit parameters lead to approximately sinusoidal circuit out- 
puts: 



V m = 0.75 V, 
R L = 200 ft, 



V C c=5V, V EE 
L = 100 x 10" 6 H, 



-5V, R E e = 100ft, 
/3 F = 80, Ron = H5ft, 



Ci = 45 x 10" y F, C 2 = 58 x 10" y F. 



(10.155) 
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Figure 10.22 Sinusoidal operations (a) phase portrait of Colpitts operator and 
(b) collector-emitter voltage, showing typical results of applying MATLAB routine ode23 
to the simulation of the Colpitts oscillator circuit. Equation (10.155) specifies the circuit 
parameters for the results shown here. 



Figure 10.22 shows the phase portrait for the oscillator using these parameter val- 
ues. We see that vce(0 is much more sinusoidal than in Fig. 10.21. The trajectory 
in the phase portrait of Fig. 10.22 is tending to an elliptical closed curve indicative 
of simple harmonic (i.e., sinusoidal) oscillation. 



10.6 STIFF SYSTEMS 

Consider the general system of coupled first-order ODEs 
dxQ(t) 



dt 

dx\{t) 

dt 



./b(xo,xi, . . . ,x m _i, t), 
/i(x ,xi, ...,x m -i,t), 



(10.156) 



dXm-l(t) 

dt 



— fm-l(XQ,Xl, ...,X m -l,t), 
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which we wish to solve for t > given x(0), where x(t) = [x$(t) x\(t) 
■■■ x m -i(t)] T . If we also define f(x(t),t) — [fo(x(t),t) fi(x(t),t) ■■■ f, 



m — l 



(x(t), t)] , then we may express (10.156) in compact vector form as 



dx(t) 
dt 



= f(x(t), t). 



(10.157) 



We have so far described a general order m ODE IVP. 

If we now wish to consider the stability of a numerical method applied to the 
solution of (10.156) [or (10.157)], then we need to consider the model problem 



dx(t) 
dt 



Ax(t), 



(10.158) 



where 



3/o(*(0), 0) 

3/i(*(0),0) 

3xo 



3/o(*(0),0) 

dx\ 
3/i(s(0),0) 

3xi 



3/o (3c (0), 0) 

3x m _i 
3/i(x(0),0) 



3x 



m—\ 



3/ m _i(x(0),0) 3/ m _i(x"(0),0) 



3/ m _!(x-(0),0) 



(10.159) 



3xo 3xi 3x m -i 

[which generalizes A e R 2x2 in (10.94)]. The solution to (10.158) is given by 

x(f) = e At x(0) (10.160) 

[recall (10.92)]. 9 Ensuring the stability of the order m linear ODE system in 
(10.158) requires all the eigenvalues A.& of A to have negative real parts (i.e., 
Re[A / ( : ] < for all k = 0, 1, . . . , m — 1, where X^ is the kth eigenvalue of A). 
Of course, if m — 1, then with x(t) — xo(t), (10.158) reduces to 



dx(t) 
dt 



Xx{t), 



(10.161) 



which is the model problem (10.28) again. Recall once again Example 10.5, for 
which we found that 



k = 



3/C*o, t ) 

3x 



2 
to' 



Note that if we know x(tQ) (any tQ s R), then we may slightly generalize our linear problem (10.158) 
to determining x(t) for all ( > tQ, in which case 

x(t) = e A{t -'^x(to) 

replaces (10.160). However, little is lost by assuming /q = 0. 
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as (10.41) is the solution to dx/dt — t 2 — 2x/t for t > to > 0. If ?o is small, then 
\X\ is large, and we saw that numerical methods, especially low-order explicit 
ones, had difficulty in estimating x(t) accurately when t was near fn- If we recall 
(10.33) (which stated that h < 2/\X\) as an example, we see that large negative 
values for X force us to select small step sizes h to ensure stability of the explicit 
Euler method. Since k is an eigenvalue of A — [X] in (10.161), we might expect 
that this generalizes. In other words, a numerical method can be expected to have 
accuracy problems if A in (10.159) has eigenvalues with large negative real parts. 
In a situation like this x(t) in (10.156) has (it seems) a solution that changes so 
rapidly for some time intervals (e.g., fast startup transients) that accurate numerical 
solutions are hard to achieve. Such systems are called stiff systems. 

So far our definition of a stiff system has not been at all rigorous. Indeed, 
a rigorous definition is hard to come by. Higham and Trefethen [15] argue that 
looking at the eigenvalues of A alone is not enough to decide on the stiffness of 
(10.156) in a completely reliable manner. It is possible, for example, that A may 
have favorable eigenvalues and yet (10.156) may still be stiff. 

Stiff systems will not be discussed further here except to note that implicit 
methods, or higher-order predictor-corrector methods, should be used for their 
solution. The paper by Higham and Trefethen [15] is highly recommended reading 
for those readers seriously interested in the problems posed by stiff systems. 

10.7 FINAL REMARKS 

In the numerical solution (i.e., simulation) of ordinary differential equations (ODEs), 
two issues are of primary importance: accuracy and stability. The successful sim- 
ulation of any system requires proper attention to both of these issues. 

Computational efficiency is also an issue. Generally, we prefer to use the largest 
possible step size consistent with required accuracy, and as such to avoid any 
instability in the simulation. 

APPENDIX 10.A MATLAB CODE FOR EXAMPLE 10.8 



% f23.m 

% This defines function f (x,t) in the differential equation for Example 10. 
% (in Section 10.2) . 



function y = f23(x,t) 
y = t*t - (2*x/t); 

% Runge.m 

% This routine simulates the Heun's, and 4th order Runge-Kutta methods as 
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% applied to the differential equation in Example 10.8 (Sect. 10.2), so this 
% routine requires function f23.m. It therefore generates Fig. 10.9. 

function Runge 

to = .05; % initial time (starting time) 
xO =1.0; % initial condition (x(tO)) 

% Exact solution x(t) 

c = (5*t0*t0*x0) - (t0"5); 
te = [tO: .02:1 .5] ; 
for k = 1 :length(te) 

xe(k) = (te(k)*te(k)*te(k))/5 + c/(5*te(k)*te(k) ) ; 

end; 

h = .05; 

% Heun's method simulation 

xh(1) = xO; 
th(1) = tO; 
for n = 1 :25 

fn = th(n)*th(n) - (2*xh(n)/th(n) ) ; %f(x_n,t_n) 

th(n+1) = th(n) + h; 

xn1 = xh(n) + h*fn; 

tn1 = th(n+1) ; 

fn1 = tn1*tn1 - (2*xn1/tn1); % f (x_{n+1},t_{n+1}) (approx.) 

xh(n+1) = xh(n) + (h/2)*(fn + fn1); 

end; 

% 4th order Runge-Kutta simulation 

xr(1 ) = xO; 
tr(1) = tO; 
for n = 1 :25 

t = tr(n); 

x = xr(n) ; 

k1 = f23(x,t); 

k2 = f23(x + .5*h*k1,t + .5*h); 

k3 = f23(x + .5*h*k2,t + .5*h); 

k4 = f23(x + h*k3,t + h); 

xr(n+1) = xr(n) + (h/6)*(k1 + 2*k2 + 2*k3 + k4) ; 

tr(n+1) = tr(n) + h; 

end; 

plot(te,xe, ' - ' ,tr,xr, ' - -o' ,th,xh, ' --+' ) , grid 

legend('x(t) ' , '4th Order Runge-Kutta (h = .05)','Heun (h = .05)',1); 

xlabel(' Time (t) ') 

ylabel(' Amplitude ') 



APPENDIX 10.B MATLAB CODE FOR EXAMPLE 10.13 

% 

% fR.m 

% 
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% This is Equation (10.15) of Chapter 10 pertaining to Example 10.2. 

function i = fR(v) 

VTH = 0.75; % Threshold voltage in volts 

RON = 100; % On resistance of NPN BJT Q in Ohms 

if v <= VTH 

i = 0; 
else 

i = (v-VTH)/R0N; 

end; 

% vCC.m 

% Supply voltage function v_CC(t) for Example 10.2 of Chapter 10. 
% Here v_CC(t) = V_CC u(t) (i.e., oscillator switches on at t = 0) . 



function v = vCC(t) 

VCC = 5; 
if t < 

v = 0; 
else 

v = VCC; 

end; 

% Colpitts.m 

% Computes the right-hand side of the state equations in Equation (10.17a,b,c) 
% pertaining to Example 10.2 of Chapter 10. 



function y = Colpitts(t,x) 

C1 = 54e-9; 
C2 = 54e-9; 
REE = 400; 
VEE = -5; 
betaF = 200; 
RL = 35; 
L = 98.5e-6; 

y(1) = ( x(3) - betaF*fR(x(2)) )/C1; 

y(2) = ( -(x(2)+VEE)/REE - fR(x(2)) - x(3) )/C2; 

y(3) = ( vCC(t) - x(1) + x(2) - RL*x(3))/L; 

y = y- '; 

% SimulateColpitts.m 

% This routine uses vCC.m, fR.m and Colpitts.m to simulate the Colpitts 
% oscillator circuit of Example 10.2 in Chapter 10. It produces 
% Figure 10.21 in Chapter 10. 

% The state vector x(:,:) is as follows: 
% x(:,1) = v_CE(t) 
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% x(:,2) = v_BE(t) 

% x(:,3) = i_L(t) 



function SimulateColpitts 

[t,x] = ode23(@Colpitts, [ 0.003 ], [000]); 

% [ 0.003 ] ---> Simulate to 3 milliseconds 
% [ ] ---> Initial state vector 

elf 

L = length(t) ; 

subplot(211), plot(x(:,1),x(:,2)), grid 

xlabelC v_{CE} (t) (volts) ') 

ylabelf v_{BE} (t) (volts) ') 

title(' Phase Portrait of the Colpitts Oscillator (Chaotic Regime) 

subplot(212), plot(t(L-1999:L),x(L-1999:L,1),'-'), grid 

xlabel(' t (seconds) ') 

ylabelf v_{CE} (t) (volts) ') 

title (' Collector-Emitter Voltage (Chaotic Regime) ') 
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PROBLEMS 

10.1. Consider the electric circuit depicted in Fig. 10.P.1. Find matrix A e R 2x2 
and vector b e R 2 such that 



dt 

dJL 2 (t) 
dt 









MO 


= A 






lL 2 (t) 







™ s (t), 



where iL k (t) is the current through inductor L^ (k e {1,2, 3}). 

(Comment: Although the number of energy storage elements in the circuit is 
3, there are only two state variables needed to describe the circuit dynamics.) 

10.2. The circuit in Fig. 10.P.2 is a simplified model for a parametric amplifier. 
The amplifier contains a reverse-biased varactor diode that is modeled by 
the parallel interconnection of linear time-invariant capacitor Co and linear 
time-varying capacitor C(t). You may assume that C(t) = 2C\ cos(a> p t), 
where C\ is constant, and co p is the pumping frequency . Note that 

ic(t) = —\C(t)v(t)l 

dt 

The input to the amplifier is the ideal cosinusoidal current source i s {t) = 
2I S cos(a>ot), and the load is the resistor R, so the output is the current /(f) 



Ufl 



uo 




Figure 10.P.1 The linear electric circuit for Problem 10.1. 
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l,(t) 





WO 




Load 



~V~ 



Varactor diode model 
Figure 10.P.2 A model for a parametric amplifier (Problem 10.2). 



into R. Write the state equations for the circuit, assuming that the state vari- 
ables are v(t), and i'i(f). Write these equations in matrix form. (Comment: 
This problem is based on the example of a parametric amplifier as consid- 
ered in C. A. Desoer and E. S. Kuh, Basic Circuit Theory, McGraw-Hill, 
New York, 1969.) 

10.3. Give a detailed derivation of Eqs. (10.17). 

10.4. The general linear first-order ODE is 



dx(t) 
dt 



= a(t)x(t) + b(t), x(to) = xq. 



Use the trapezoidal method to find an expression for x n +\ in terms of x n , 
t„, and f„+i- 

10.5. Prove that the trapezoidal method for ODEs is unconditionally stable. 

10.6. Consider Theorem 10.2. Assume that jtrj = x(to). Use (10.23) to find an 
upper bound on \x(t„) — x n \/M for the following ODE IVPs: 

(a) — = 1 -2x, x(0) = 1. 
dt 

dx 

(b) — = 2cosx, x(0) = 0. 
dt 

10.7. Consider the ODE IVP 



dx(t) 
dt 



= \-2x, x(0) = l. 



(a) Approximate the solution to this problem using the explicit Euler 
method with h = 0.1 for n = 0, 1, . . . , 10. Do the computations with a 
pocket calculator. 

(b) Find the exact solution x(t). 
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10.8. Consider the ODE IVP 

dx(t) 



dt 



2cos:t, x(0) = 0. 



(a) Approximate the solution to this problem using the explicit Euler 
method with h — 0.1 for n = 0, 1, . . . , 10. Do the computations with a 
pocket calculator. 

(b) Find the exact solution x(t). 

10.9. Write a MATLAB routine to simulate the ODE 

dx x 

dt x + t 

for the initial condition jc(0) = 1. Use both the implicit and explicit Euler 
methods. The program must accept as input the step size h, and the number 
of iterations N that are desired. Parameters h and N are the same for both 
methods. The program output is be written to a file in the form of a table 
something such as (e.g., for h — 0.05, and N — 5) the following: 



time step 


ex| 


plicit 


x_n 


implicit x 


.0000 




value 




value 


.0500 




value 




value 


.1000 




value 




value 


.1500 




value 




value 


.2000 




value 




value 


.2500 




value 




value 


Test your program 


out on 










h = 


0.0L 


N = 100 



and 

ft = 0.10, N =10. 



10.10. Consider 

dr(t\ 

a(l -2pt z )e- pt . (10.P.1) 



dx(t) „ nn2 ,-pt 2 



dt 

(a) Verify that for t > 0, with x(0) = 0, we have the solution 

x(t) = ate- p, \ (10.P.2) 

(b) For what range of step sizes h is the explicit Euler method a stable 
means of solving (10.P.1)? 
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(c) Write a MATLAB routine to simulate (1 O.P.I) for x(0) = using both 
the explicit and implicit Euler methods. Assume that a — 10 and P — I. 
Test your program out on 

ft = 0.01, N=400 (10.P.3a) 

and 

h = 0.10, N = 40. (10.P.3b) 

The program must produce plots of [x n \n = 0, I, . . . , N] (for both 
explicit and implicit methods), and x(t) (from (10.P.2)) on the same 
graph. This will lead to two separate plots, one for each of (10.P.3a) 
and (10.P.3b). 

10.11. Consider 

dx(t) 

— — =2tx + l, x(0) = 0. 
dt 

(a) Find \x n \n = 0, 1, . . . , 10} for h = 0.1 using the explicit Euler method. 
Do the calculations with a pocket calculator. 

(b) Verify that 



— e I e 

Jo 



x{t) — e'~ I e s ds. 

'o 



(c) Does the stability condition (10.33) apply here? Explain. 

10.12. Prove that Runge-Kutta methods of order one have no degrees of freedom 
(i.e., we are forced to select c\ in only one possible way). 

10.13. A fourth-order Runge-Kutta method is 

x„ + i — x n + \h\k\ + 2&2 + 2&3 + £4] 
for which 

h = f(x„, t n ), k 2 = f \x„ + \hk\,t n + jhj , 

kj = / [x n + hhlc2, t n + Ih I , k/\ — f(x n + hie?,, t n + h). 

When applied to the model problem, we get x n — cr n XQ for n e Z + . Derive 
the expression for a . 

10.14. A third-order Runge-Kutta method is 

x„+i = x n + \\ki + Ak 2 + k 3 ] 
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for which 

k\ = hf{x n , t n ), k 2 = hf(x„ + \h, t n + \k\), 
kj, — hf(x n + h, t n — k\ + Iki). 

(a) When applied to the model problem, we get x n — a n xo for n e Z + . 
Derive the expression for a . 

(b) Find the allowable range of step sizes h that ensure stability of the 
method. 

10.15. Consider the fourth-order Runge-Kutta method in Eqs. (10.80) and (10.81). 
Show that if f(x, t) = f(t), then the method reduces to Simpson's rule for 
numerical integration over the interval [t n , t n +\\. 

10.16. Recall Eq. (10.98). Suppose that A e R 2x2 has distinct eigenvalues yk such 
that Re[y^] > for at least one of the eigenvalues. Show that / + hA will 
have at least one eigenvalue \k such that 11^1 > 1. 

10.17. Consider the coupled first-order ODEs 

dx 



= -yJx 2 + y 2 , (10.P.4a) 

dt 

dy 



= xJx 2 + y 2 , (10.P.4b) 

dt v 

where (xo, yo) — (*(0), y(0)) are the initial conditions. 

(a) Prove that for suitable constants ro and 9q, we have 

x(t) = r cos(r t + O ), y (t) = r Q sin(r Q t + 9 Q ) . (10.P.5) 

(b) Write a MATLAB routine to simulate the system represented by (10.P.4) 
using the explicit Euler method [which will produce x n — [x n y n ] T 
such that x n « x(t n ) and y„ « y(t n )]. Assume that h — 0.05 and the 
initial condition is Jo = [1 0] r . Plot x„ and (x(t), y(t)) (via (10.P.5)) 
on the (x, y) plane. 

(c) Write a MATLAB routine to simulate the system represented by (10.P.4) 
using Heun's method [which will produce x n — [x n y n ] T such that 
x n »s x(t n ) and y„ « y(t„)]. Assume that h — 0.05 and the initial con- 
dition is Jo = [1 0] T . Plot x„, and (x(t), y(t)) [via (10.P.5)] on the 
(x, y) plane. 

Make reasonable choices about the number of time steps in the simulation. 

10.18. In the previous problem the step size is h — 0.05. 

(a) Determine whether the simulation using the explicit Euler method is 
stable for this choice of step size. {Hint: Recall that one must consider 
the eigenvalues of / + hA.) 
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(b) Determine whether the simulation using Heun's method is stable for 
this choice of step size. [Hint: Consider the implications of (10.121).] 

Use the MATLAB eig function to assist you in your calculations. 

10.19. A curve in R 2 is specified parametrically according to 

x(t) — Acos(cot), y(t) = aAcos(a>t — <j>), (10.P.6) 

where a, A > 0, and t e R is the "parameter." When the points (x(t), y(t)) 
are plotted, the result is what electrical engineers often call a Lissajous figure 
(or curve), which is really just an alternative name for a phase portrait. 

(a) Find an implicit function expression for the curve, that is, find a descrip- 
tion of the form 

fix, y) = (10.P.7) 

[i.e., via algebra and trigonometry eliminate t from (10.9.6) to obtain 
(10.9.7)]. 

(b) On the (x, y) plane sketch the Lissajous figures for cases = 0, </> = 
±| and <p = ±|. 

(c) An interpretation of (10.9.6) is that x(t) may be the input voltage from 
a source in an electric circuit, while y(t) may be the output voltage 
drop across a load element in the circuit (in the steady-state condition 
of course). Find a simple expression for sin<^> in terms of B such that 
/(0, B) — [i.e., point(s) B on the y axis where the curve cuts the 
y axis], and in terms of a and A. {Comment: On analog oscilloscopes 
of olden days, it was possible to display a Lissajous figure, and so use 
this figure to estimate the phase angle (f> on the lab bench.) 

10.20. Verify the values for Xk in Table 10.1 for m — 2. 

10.21. Verify the values for Xk in Table 10.2 for m — 2. 

10.22. For the ODE IVP 

dx(t) 



dt 



f(x,t), x(f )=x , 



write pseudocode for a numerical method that approximates the solution to 
it using an AB method for m — 2 as a predictor, an AM method for m — 2 
as a corrector, and the third-order Runge-Kutta (RK) method from Problem 
10.14 to perform the initialization. 

10.23. From Forsythe, Malcolm, and Moler (see Ref. 5 in Chapter 2) 

— = 998x + 1998y (10.P.8a) 

dt 

-1- = -999* - 1999y (10.P.8b) 

dt 
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has the solution 

x(t) = 4e~' - 3e~ 1000 ', (10.P.9a) 

y(t) = -2e~ f + 3e- wm , (10.P.9b) 

where x(0) = y(0) = 1. Recall A as given by (10.94). 

(a) By direct substitution verify that (10.P.9) is the solution to (10. P. 8). 

(b) If the eigenvalues of / + hA are Xo and X\, plot \Xq\ and |Ai| versus h 
(using MATLAB). 

(c) If the eigenvalues of / + hA + \h 2 A 2 [recall (10.121)] are X and Xi, 
plot \X Q \ and |Ai| versus h (using MATLAB). 

In parts (b) and (c), what can be said about the range of values for h leading 
to a stable simulation of the system (10.P.8)? 

10.24. Consider the coupled system of first-order ODEs 

dx{t) 



dt 



= Ax(t) + y(t), (10.P.10) 



where A e R nxn and x(t), y{t) e R" for all t e R + . Suppose that the eigen- 
values of A are Xq, Aj, . . . , X n _\ such that Re[A^] < for all k e Z„. 
Suppose that 

a < Re[X k ] < x < 

for all k. A stiffness quotient is defined to be 

a 

x 

The system (10.P.10) is said to be stiff ii r ^> 1 (again, we assume Re[A^] < 
for all k). 

For the previous problem, does A correspond to a stiff system? (Comment: 
As Higham and Trefethen [15] warned, the present stiffness definition is 
not entirely reliable.) 
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Numerical Methods 
for Eigenproblems 



11.1 INTRODUCTION 

In previous chapters we have seen that eigenvalues and eigenvectors are important 
(e.g., recall condition numbers from Chapter 4, and the stability analysis of numer- 
ical methods for ODEs in Chapter 10). In this chapter we treat the eigenproblem 
somewhat more formally than previously. We shall define and review the basic 
problem in Section 1 1 .2, and in Section 11.3 we shall apply this understanding to 
the problem of computing the matrix exponential [i.e., exp(Af), where A e R" x " 
and t € R] since this is of central importance in many areas of electrical and com- 
puter engineering (signal processing, stability of dynamic systems, control systems, 
circuit simulation, etc.). In subsequent sections we will consider numerical methods 
to determine the eigenvalues and eigenvectors of matrices. 



11.2 REVIEW OF EIGENVALUES AND EIGENVECTORS 

In this section we review some basic facts relating to the determination of eigen- 
values and eigenvectors of matrices. Our emphasis, with a few exceptions, is on 
matrices that are diagonalizable. 

Definition 11.1: Eigenproblem Let A e C" x ". The eigenproblem for A is 
to find solutions to the matrix equation 

Ax = Xx, (11.1) 

where X e C and x € C" such that x ^ 0. A solution (X, x) to (11.1) is called an 
eigenpair, X is an eigenvalue, and x is its corresponding eigenvector. 

Even if A € R nxn (the situation we will emphasize most) it is very possible to 
have X € C and x e C" . We must also emphasize that x — is never permitted to 
be an eigenvector for A. 

We may rewrite (11.1) as (/ is the n x n identity matrix) 

(A-XI)x = 0, (11.2a) 
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or equivalently as 



(XI - A)x = 0. 



(11.2b) 



Equations (11.2) are homogeneous linear systems of n equations in n unknowns. 
Since x — is never an eigenvector, an eigenvector x must be a nontrivial solution 
to (11.2). An n x n homogeneous linear system has a nonzero (i.e., nontrivial) 
solution iff the coefficient matrix is singular. Immediately, eigenvalue X satisfies 



det(A - XI) = 0, 



(11.3a) 



or equivalently 



det(XI - A) = 0. 



(11.3b) 



Of course, p(X) — det(A/ — A) is a polynomial of degree n. In principle, we 
may find eigenpairs by finding p(X) (the characteristic polynomial of A), then 
finding the zeros of p(X), and then substituting these into (11.2) to find x. In 
practice this approach really works only for small analytical examples. Properly 
conceived numerical methods are needed to determine eigenpairs reliably for larger 
matrices A. 

To set the stage for what follows, consider the following examples. 

Example 11.1 From Hill [1], consider the example 



Here p(X) — det(XI — A) — (X — 1) (X — 3) so A has a double eigenvalue (eigen- 
value of multiplicity 2) at X — 1, and a simple eigenvalue at X = 3. The eigenvalues 
may be individually denoted as Xo — l,X\ — 1, and A 2 — 3. 
For X — 3, (3/ — A)x — becomes 



2 




XQ 







2 2 




Xl 


= 










X2 








and from the application of elementary row operations this reduces to 



1 




XQ 







1 




Xl 


= 










X2 








Immediately, xq — x\ — 0, and x-i is arbitrary (except, of course, it is not allowed 
to be zero) so that the general form of the eigenvector corresponding to X — X2 is 



,(2) 



[0 x 2 ] ' e C J 
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2 














-2 


1 





" 








1 














x 




" " 




Xl 


= 







_ *2 








On the other hand, now let us consider X — Xq — X\. In this case (/ — A)x — 
becomes 



which reduces to 



Immediately, xo = X2 = 0, and x\ is arbitrary, so an eigenvector corresponding to 
X — Xo — X\ is of the general form 





xo 




" " 




XI 


= 







_ X 2 








,(0) 



= [ xi f e C 3 . 



Even though X = 1 is a double eigenvalue, we are able to find only one eigenvector 
for this case. In effect, one eigenvector seems to be "missing." 

Example 11.2 Now consider (again from Hill [1]) 






-2 


1 


1 


3 


-1 








1 



A = 



Here p(X) — det(XI — A) — (X — l) 2 (X — 2). The eigenvalues of A are thus Xo 
l,X\ — 1, and X2 — 2. 

For X — 2, (21 — A)x — becomes 



2 
1 



2 -1 

-1 1 

1 


1 





1 " 
1 






xo 




" 




Xl 


= 







_ X 2 








x 




" " 


XI 


= 





_ X 2 








which reduces to 



so the general form of the eigenvector for X — X2 is 

v (2) = x (2) = [ -x x f e C 3 
Now, if we consider X = Xq = A-i, (/ — A)x = becomes 



1 


2 


-1 




xo 







1 


-2 


1 




Xl 


= 
















X2 








TLFeBOOK 



REVIEW OF EIGENVALUES AND EIGENVECTORS 



483 



which reduces to 



1 2 


-1 




XQ 















X\ 


= 













X2 








Since xq + 2x\ — X2 — 0, we may choose any two of xq, X\, or X2 as free para- 
meters giving a general form of an eigenvector for X — Xq — X\ as 



v®> = x 



1 " 




' " 




1 


+ X\ 


1 

2 



,-(()) 



eC 3 . 



Thus, we have eigenpairs (Xq, x ( '), (X\,x ( '), (X2, x ( ') 



AV\ 



,(2) A 



We continue to emphasize that in all cases any free parameters are arbitrary, 
except that they must never be selected to give a zero-valued eigenvector. In 
Example 11.2 X = 1 is an eigenvalue of multiplicity 2, and i/ - 1 is a vector in 



a two-dimensional vector subspace of C 3 . 
X = 1 is also of multiplicity 2, and yet i/ ^ 
vector subspace of C 3 . 



On the other hand, in Example 11.1 
is only a vector in a one-dimensional 



Definition 11.2: Defective Matrix For any A & C nxn , if the multiplicity 
of any eigenvalue X e C is not equal to the dimension of the solution space 
(eigenspace) of (XI — A)x = 0, then A is defective. 

From this definition, A in Example 11.1 is a defective matrix, while A in 
Example 11.2 is not defective (i.e., is nondefective). In a sense soon to be made 
precise, defective matrices cannot be diagonalized. However, all matrices, diago- 
nalizable or not, can be placed into Jordan canonical form, as follows. 

Theorem 11.1: Jordan Decomposition If A e C" x ", then there exists anon- 
singular matrix T e C nxn such that 



T~ l AT = diag(./o, J\, 



Jk-i), 



where 



Ji 



A, 


1 


• 


• 








A, 


1 • 


• 











• 


• A; 


1 








• 


• 


X 



eC n 



and Ei=0 m i = "• 
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Proof See Halmos [2] or Horn and Johnson [3]. 

Of course, A., in the theorem is an eigenvalue of A. The submatrices 7, are called 
Jordan blocks. The number of blocks k and their dimensions m, are unique, but 
their order is not unique. Note that if an eigenvalue has a multiplicity of unity (i.e., 
if it is simple), then the Jordan block in this case is the lxl matrix consisting 
of that eigenvalue. From the theorem statement the characteristic polynomial of 
A € C" x " is given by 



k-\ 



p(X) = det(A7 - A) = Y\0- ~ *;)" 



(11.4) 



i=0 



But if A € R" x ", and if A, € C for some i, then X* must also be an eigenvalue of 
A, that is, a zero of p(X). This follows from the fundamental theorem of algebra, 
which states that complex-valued roots of polynomials with real- valued coefficients 
must always occur in complex -conjugate pairs. 

Example 11.3 Consider (with ^ kit, k e Z) 



A = 



cost 
sin 6 



- sine 
cosf 



eR 



2x2 



(2x2 rotation operator from Appendix 3. A). We have the characteristic equation 

X — cos 6 



p(X) = det(XI - A) = det ( 



smf 
X — cos ( 



= X z - 2 cos 6X + 1 = 

for which the roots (eigenvalues of A) are therefore X — e ± ' e . Define Xq — e' e , 
X\ — e~' e . Clearly X\ — Xt (i.e., the two simple eigenvalues of A are a conju- 
gate pair). 

For X — Xq, (XqI — A)x — is 



sint9 



which reduces to 



J 1 
-1 J 



-J 







XQ 

x\ 








The eigenvector for X — Xo is therefore of the form 

, a e C. 



x®> = a 
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Similarly, for k — X\, the homogeneous linear system (X\I — A)x = is 



-j 1 
-1 -j 



x\ 



which reduces to 



The eigenvector for A. = X \ is therefore of the form 



" 1 j ' 

.00 




Xq 


= 


" " 
_ _ 



x m = b 



beC. 



Of course, free parameters a, and b are never allowed to be zero. 

Computing the Jordan canonical form when m, > 1 is numerically rather dif- 
ficult (as noted in Golub and Van Loan [4] and Horn and Johnson [3]), and so 
is often avoided. However, there are important exceptions often involving state- 
variable (state-space) systems analysis and design (e.g., see Fairman [5]). Within 
the theory of Jordan forms it is possible to find supposedly "missing" eigenvectors, 
resulting in a theory of generalized eigenvectors. We will not consider this here as 
it is rather involved. Some of the references at the end of this chapter cover the 
relevant theory [3]. 

We will now consider a series of theorems leading to a sufficient condition for 
A to be nondefective. Our presentation largely follows Hill [1]. 

Theorem 11.2: If A € C" x ", then the eigenvectors corresponding to two dis- 
tinct eigenvalues of A are linearly independent. 

Proof We employ proof by contradiction. 

Suppose that (a, x) and {fi, y) are two eigenpairs for A, and a ^ fi. Assume 
y — ax for some a # (a e C). Thus 



and also 



Hence 



implying that 



fly = Ay — aAx = aax 
/3y = a/3x. 



apx — aax, 



a{f5 — a)x — 0. 
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But x ^ as it is an eigenvector of A, and also a ^ 0. Immediately, a — j3, 
contradicting our assumption that these eigenvalues are distinct. Thus, y — ax is 
impossible; that is, we have proved that y is independent of x. 

Theorem 11.2 leads us to the next theorem. 

Theorem 11.3: If A € C nx " has n distinct eigenvalues, then A has n linearly 
independent eigenvectors. 

Proof Uses mathematical induction (e.g., Stewart [6]). 

We have already seen the following theorem. 

Theorem 11.4: If A € R" x " and A = A T , then all eigenvalues of A are real- 
valued. 

Proof See Hill [1], or see the appropriate footnote in Chapter 4. 

In addition to this theorem, we also have the following one. 

Theorem 11.5: If A € R" x ", and if A = A T , then eigenvectors corresponding 
to distinct eigenvalues of A are orthogonal. 

Proof Suppose that (a, x) and (ft, y) are eigenpairs of A with a ^ /3. We wish 
to show that x T y — y T x — (recall Definition 1.6). Now 



so that 



ax — Ax = A x 



ay T x — y T A T x — (Ay) T x = fiy T x, 



implying that 

(a - fi)y T x = 0. 

But a ^ p so that y T x — 0, that is, x ± y. 

Theorem 11.5 states that eigenspaces corresponding to distinct eigenvalues of a 
symmetric, real-valued matrix form mutually orthogonal vector subspaces of R". 
Any vector from one eigenspace must therefore be orthogonal to any eigenvector 
from another eigenspace. If we recall Definition 11.2, it is apparent that all sym- 
metric, real-valued matrices are nondefective, // their eigenvalues are all distinct. ' 
In fact, even if A € C" x " and is not symmetric, then, as long as the eigenvalues 
are distinct, A will be nondefective (Theorem 11.3). 

It is possible to go even further and prove that any real-valued, symmetric matrix is nondefective, 
even if it possesses multiple eigenvalues. Thus, any real-valued, symmetric matrix is diagonalizable. 
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Definition 11.3: Similarity Transformation If A, B € C nxn , and there is a 
nonsingular matrix P e C nxn such that 

B = P~ l AP, 

we say that B is similar to A, and that P is a similarity transformation. 

If A e C nxn , and A has n distinct eigenvalues forming n distinct eigenpairs 
{{k k , x {k) )\k = 0, 1, . . . , n - 1}, then 



allows us to write 



Al/V"...^-"] 



= p 



Ax {k) = ***<« 








r A 


•■ 


> (P) JC (i)... Jt (i,-i) ] 





ki ■■ 


=i> 





•■ 






k n - 



(11.5) 
that is, AP — PA, where A = diag(Ao, k\, . . . , k n -\) e C" x " is the diagonal matrix 
of eigenvalues of A. Thus 

P~ l AP=A, (11.6) 

and the matrix of eigenvectors P e C" x " of A defines the similarity transformation 
that diagonalizes matrix A. More generally, we have the following theorem. 

Theorem 11.6: If A, B e C" x ", and A and B are similar matrices, then A and 
Z? have the same eigenvalues. 

Proof Since A and B are similar, there exists a nonsingular matrix P e C" x " 
such that B — P~ l AP. Therefore 

det(kl - B) = det(AJ - P _1 AF) 

= det(P" 1 (PP" 1 A- A)P) 

= det(P _1 ) det(kl - A) det(P) 

= det(A7 - A). 

Thus, A and B possess the same characteristic polynomial, and so possess identical 
eigenvalues. 

In other words, similarity transformations preserve eigenvalues. 2 Note that The- 
orem 11.6 holds regardless of whether A and B are defective. In developing (11.6) 

This makes such transformations highly valuable in state-space control systems design, in addition to 
a number of other application areas. 
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we have seen that if A € C nxn has n distinct eigenvalues, it is diagonalizable. We 
emphasize that this is only a sufficient condition. Example 11.2 confirms that a 
matrix can have eigenvalues with multiplicity greater than one, and yet still be 
diagonalizable. 



11.3 THE MATRIX EXPONENTIAL 

In Chapter 10 the problem of computing e At (A e R" x ", and t e R) was associated 

with the stability analysis of numerical methods for systems of ODEs. It is also 

noteworthy that to solve 

dx(t) 

-^-± = Ax{t)+by(t) (11.7) 

at 

[x(t) = [x (t)xi(t) ■ ■ -x n _i(t)] T e R", A e R" xn , and b e R" with y(t) e R for 
all t] required us to compute e At [recall Example 10.1, which involved an example 
of (1 1.7) from electric circuit analysis; see Eq. (10.9)]. Thus, we see that computing 
the matrix exponential is an important problem in analysis. In this section we shall 
gain more familiarity with the matrix exponential because of its significance. 

Moler and Van Loan [7] caution that computing the matrix exponential is a 
numerically difficult problem. Stable, reliable, accurate, and computationally effi- 
cient algorithms are not so easy to come by. Their paper [7], as its title states, 
considers 19 methods, and none of them are entirely satisfactory. Indeed, this 
paper [7] appeared in 1978, and to this day the problem of successfully computing 
e At for any A e C nxn has not been fully resolved. We shall say something about 
why this is a difficult problem later. 

Before considering this matter, we shall consider a general analytic (i.e., hand 
calculation) method for obtaining e At for any A € R" x ", including when A is 
defective. In principle, this would involve working with Jordan decompositions 
and generalized eigenvectors, but we will avoid this by adopting the approach 
suggested in Leonard [8]. 

The matrix exponential e Al may be defined in the expected manner as 

°° 1 
O(f) = e At = V— A k t k , (11.8) 

*— ' kl 

so, for example, the kth derivative of the matrix exponential is 

0>«(f) = A k e At = e At A k (11.9) 

for k e Z + (<J>(°)(f) = <t>(r)). To see how this works, consider the following special 
case k = 1: 

<D (1) (f) = — I V — A k t k \ = — \l + —At + —A 2 t 2 H h — A k t k + ■ ■ 

w dt\f^k\ j dt 1 1! 2! k\ 
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1 2 , 
= — A+ —A 2 t + - 
1! 2! 


. .-1 A *f* 1 h 


= a\i+ —At + -- 
1 1! 


1 A *-i,*-i 
(t-1)! 


= Ae At = e A, A. 





It is possible to formally verify that the series in (11.8) converges to a matrix 
function of t e R by working with the Jordan decomposition of A. However, 
we will avoid this level of detail. But we will consider the situation where A 
is diagonalizable later on. 

There is some additional background material needed to more fully appreciate 
[8], and we will now consider this. The main result is the Cayley-Hamilton theorem 
(Theorem 11.8, below). 

Definition 11.4: Minors and Cofactors Let A e C" x ". The minor mij is the 
determinant of the (n — 1) x (n — 1) submatrix of A derived from it by deleting 
row i and column j . The cofactor en associated with m,y is en — (—l) l+ 'mij for 
all i, j e Z„. 

A formula for the inverse of A (assuming this exists) is given by Theorem 11.7. 

Theorem 11.7: If A e C" x " is nonsingular, then 

A_1 = XT77T ad K A )' ( 1L1 °) 

det(A) 

where adj(A) (adjoint matrix of A) is the transpose of the matrix of cofactors of 
A. Thus, if C = [cjj] € C nxn is the matrix of cofactors of A, then adj(A) = C T . 

Proof See Noble and Daniel [9]. 

Of course, the method suggested by Theorem 11.7 is useful only for the hand 
calculation of low-order (small n) problems. Practical matrix inversion must use 
ideas from Chapter 4. But Theorem 11.7 is a very useful result for theoretical 
purposes, such as obtaining the following theorem. 

Theorem 11.8: Cayley-Hamilton Theorem Any matrix A € C" xn satisfies 
its own characteristic equation. 

Proof The characteristic polynomial for A is p(X) — det(XI — A), and can be 
written as 

p(X) — X" + a x X n ~ x H h a n -\X + a„ 

for suitable constants a^ e C. The theorem claims that 

A"+aiA" _1 H hfl„-iA + a„7 = 0, (H.H) 
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where I is the order n identity matrix. To show (11.11), we consider adj(/x/ — A) 
whose elements are polynomials in /x of a degree that is not greater than n — 1, 
where ji is not an eigenvalue of A. Hence 

adj(/x/ - A) = Mo/x"" 1 + M^"" 2 H h M n _ 2 /x + M„_i 

for suitable constant matrices M& € C" x ". Via Theorem 11.7 

(/x/ - A) adj(/x/ - A) = det(/x/ - A)/, 

or in expanded form, this becomes 

(jil - A)(M /x"- 1 + Mj/x"" 2 + • • • + M„_2M + M n _0 
= (At"+ai/x"" 1 H ho„)/. 

If we now equate like powers of /x on both sides of this equation, we obtain 

M = /, 
Mi — AMo = a\I, 
M 2 — AM\ — a 2 I, 

: (11.12) 

M„_! -AM„_ 2 = a,_i/, 
-AM„_] = a„7. 

Premultiplying 3 the jth equation in (11.12) by A n ~ J (j — 0, 1, . . . , «), and then 
adding all the equations that result from this, yields 

A" M + A" _1 (Mi - AMo) + A"~ 2 (M 2 - AM\) H h A(M„_i - AM n - 2 ) 

- AM n _i = A"+a 1 A"" 1 +a 2 A n ~ 2 H ha^iA"" 1 +a„7. 

But the left-hand side of this is seen to be zero because of cancellation of all the 
terms, and (11.11) immediately results. 

As an exercise the reader should verify that the matrices in Examples 1 1.1-1 1.3, 
all satisfy their own characteristic equations. 

We will now consider the approach in Leonard [8], who, however, assumes that 
the reader is familiar with the theory of solution of nth-order homogeneous linear 
ODEs in constant coefficients 

x (n \t) + c n - lX {n ~ l \t) + ■■■ + dx m (t) + CQx(t) = 0, (11.13) 

This means that we must multiply on the left. 
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where the initial conditions are known. In particular, the reader must know that if 
X is a root of the characteristic equation 

X n + c n -iX n - 1 + ■ ■ ■ + ciX + co = 0, (11.14) 

then if X has multiplicity m, its contribution to the solution of the initial- value 
problem (IVP) (11.13) is of the general form 

(a + a x t H h a m _it m ~ l )e Xt . (11.15) 

These matters are considered by Derrick and Grossman [10] and Reid [11]. We 
shall be combining these facts with the results of Theorem 11.10 (below). 

Leonard [8] presents two theorems that relate the solution of (11.13) to the 
computation of <!>(?) = e At . 

Theorem 11.9: Leonard I Let A e R nxn be a constant matrix with charac- 
teristic polynomial 

p(X) = det(XI - A) = X" + c n -iX n ~ l -\ h c x X + c . 

0(?) = e At is the unique solution to the nth-order matrix differential equation 

$ (n) (f) + c-i^" -1 ^*) + • • • + Cl <D (1) (f) + c <t>(0 = (11.16) 

with initial conditions 

d>(0) = /, <J> (1) (0) = A, . . . , d> ( "- 2) (0) = A n ~ 2 , O ( " _1) (0) = A"' 1 , (11.17) 

Proof We will demonstrate uniqueness of the solution first of all. 

Suppose that <t>i(t) and 02(f) are two solutions to (11.16) for the initial condi- 
tions stated in (11.17). Let <J>(f) = <l>i(£) — <J>2(0 f° r present purposes, in which 
case <J>(f) satisfies (11.16) with the initial conditions 

d>(0) = <D (1) (0) = • • • = <D ( "" 2) (0) = $ ( " -1) (0) = 0. 

Consequently, each entry of the matrix <£(f) satisfies a scalar IVP of the form 

x {n \t) + c n -ix (n - l) (t) + ■■■ + cix (1) (0 + c Q x(t) = 0, 
x(0) = x (1) (0) = • • • = x ( "" 2) (0) = x (n - l) (0) = 0, 

where the solution is x(t) — for all t, so that <$>(t) — for all t e R + . Thus, 
<t>i(t) = <t>2(£X an d so the solution must be unique (if it exists). 
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Now we confirm that the solution is <t>(?) = e At (i.e., we confirm existence in 
a constructive manner). Let A be a constant matrix of order n with characteristic 
polynomial p(X) as in the theorem statement. If now <£(f) = e At , then we recall that 

$(*)(,) = A k e At ,k = 1,2, ...,n (11.18) 

[see (11.9)] so that 

<D (n) (0 + c-i^"- 1 ^*) + • • • + ci<t> (1) (f) + co*(0 
= [A" + Cn-iA"' 1 + ■ ■ ■ + a A + c I]e At 
= p(A)e A ' = 

via Theorem 11.8 (Cayley-Hamilton). From (11.18), we obtain 

4>(°)(0) = /, O (1) (0) = A, . , . , O ( "- 2) (0) = A n ~ 2 , $ ( " _1) (0) = A"' 1 , 

and so <t>(r) = e At is the unique solution to the IVP in the theorem statement. 

Theorem 11.10: Leonard II Let A e R" x " be a constant matrix with char- 
acteristic polynomial 

p{X) = X n + C,,-!!"" 1 + • • • + cik + co, 

then 

e A> = x (0/ + xi (0 A + x 2 0) A 2 H h jc„_i (t) A"" 1 , 

where x^(f), k e Z n are the solutions to the nth-order scalar ODEs 

x {n \t) + c n -ix (n - l \t) + ■■■ + cix m (t) + c Q x(t) = 0, 
satisfying the initial conditions 

x { k j) (0) = Sj- k 
forj,keZ n {xf\t) = x k {t)). 

Proof Let constant matrix A have characteristic polynomial p(X) as in the 
theorem statement. Define 

O(f) = x (t)I + xi(t)A + x 2 (t)A 2 + ■■■+ x n -i(t)A n -\ 

where Xk(t), k e Z n are unique solutions to the «th-order scalar ODEs 

x {n \t) + c n -ix {n - l \t) + ■■■ + c { x m (t) + c x(t) = 0, 
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satisfying the initial conditions stated in the theorem. Thus, for all t e R + 
0> (n) (0 + c„-i$ ( " _1) (f) + • • • + C1 <D (1) (0 + c $(f) 

n-\ 

= J2 H n) w + c»-i4" _1) (o + • • • + ci4 1} ( ? ) + co**(o] a* 

)t=0 

= • / + • A H h • A"" 1 = 0. 

As well we see that 

$(0) = xo(0)I+xi(0)A + --- + x n -i(.0)A n - 1 =/, 

G>(1)(0) = 4 1) ( ) / + X l 1) ( () ) A +---+ JC B-l( ) A " _1 = A ' 

0(1-1) (0) = ^"-^(Q)/ + x ( "~ l) (0)A + • • • + ^"/'(O)^" 1 = A"" 1 . 

Therefore 

d>(f ) = x (?)/ + ^1 (0 A H h x„-i (t) A"" 1 

satisfies the IVP 

d> (n) (0 + c „_i* ( " _1) (0 + • • • + Cl O (1) (0 + c *(0 = 

possessing the initial conditions 

$W(0) = A* 

(A; e Z„). The solution is unique, and so we must conclude that e At — YllZo x k(t)A k 
for all t e R + , which is the central claim of the theorem. 

An example of how to apply the result of Theorem 11.10 is as follows. 

Example 11.4 Suppose that 



A = 



a y 

p 



e R 2x2 , 



which clearly has the eigenvalues X — a, p. Begin by assuming distinct eigenvalues 
for A, specifically, that a ^ p. 

The general solution to the second-order homogeneous ODE 

x (2) (t) + cix (1) (t) + c x(t) = 

with characteristic roots a, p (eigenvalues of A) is [recall (11.15)] 

x(t) — aoe°" + aie^' . 

We have x m (t) = a Q oie at + aifie pt . 
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For the initial conditions x(0) = l,x^(0) = 0, we have the linear system 
of equations 

ao + a\ — 1, 

aao + fia\ — 0, 



which solve to yield 



flo 



P-u 



, a\ 



P-a 



Thus, the solution in this case is 



x Q (t) 



p-a 



[pe at - a/']. 



Now, if instead the initial conditions are x(0) — 0, x^(0) — 1, we have the linear 
system of equations 

ao + a\ — 0, 

aao + fia\ — 1, 



which solve to yield 



1 1 

«o = , ai = - — 

P-a P- 



Thus, the solution in this case is 



x\ (t)= [- 

P — a 



o at -L. J»l 



Via Leonard II we must have 



e A 


= x Q (t)I + 


x\(t)A 








= 


1 


" Pe at - aeP> 

Pe°" - aeP' _ 






P -a 






- 


1 


~ -ae at + ae?' -ye at + ye?' ~ 
-pe°" + pe pt 




P-a 




= 


1 


' (P - a)e at -ye at + ye?' ' 
(P- a)e pt 






P-a 






= 


- g at 




J 


—a y ' 
e f>t 







(11-19) 



Now assume that a — p. 
The general solution to 



A2), 



,(D 



x w (t) + c x x w (t) + c x(t) = 
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with characteristic roots a, a (eigenvalues of A) is [again, recall (11.15)] 

x(t) — («o + a\t)e at . 

We have x^{t) — (a\ + aoa + a\at)e at . 

For the initial conditions x(0) — l,;c^(0) = we have the linear system 
of equations 

a = 1, 

a\ + aoa — 0, 

which solves to yield a\ — —a, so that 

x (t) = (1 -at)e°". 

Now if instead the initial conditions are x(0) = 0, x^(0) = 1, we have the linear 
system of equations 

«o = 0, 

a\ + aoa — 1, 
which solves to yield a\ = 1 so that 

xi (t) = te°". 
If we again apply Leonard II, then we have 

e At =xo(0/ + xi(f)A 



e at yte°" 
e at 



(11.20) 



A good exercise for the reader is to verify that x(t) — e At x(0) solves dx(t)/dt = 
Ax(t) [of course, here x(t) — [xo(t)x\(t)] T is a state vector] in both of the cases 
considered in Example 11.4. Do this by direct substitution. 

Example 11.4 is considered in Moler and Van Loan [7] as it illustrates problems 
in computing e At when the eigenvalues of A are nearly multiple. If we consider 
(11.19) when /3 — a is small, and yet is not negligible, the "divided difference" 

(11.21) 



p-a 



when computed directly, may result in a large relative error. In (11.19) the ratio 
(11.21) is multiplied by y, so the final answer may be very inaccurate, indeed. 
Matrix A in Example 11.4 is of low order (i.e., n — 2) and is triangular. This type 
of problem is very difficult to detect and correct when A is larger and not triangular. 
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Figure 11.1 An illustration of the hump phenomenon in computing e . 

Another difficulty noted in Moler and Van Loan [7] is sometimes called the hump 
phenomenon. It is illustrated in Fig. 11.1 for Eq. (11.19) using the parameters 



■1.01, 



-1.00, y=50. 



(11.22) 



Figure 11.1 is a plot of matrix 2-norm [spectral norm; recall Equation (4.37) with 
p — 2 in Chapter 4] ||e A '|l2 versus t. (It is a version of Fig. 1 in Ref. 7.) The 
problem with this arises from the fact that one way or another some algorithms for 
the computation of e At make use of the identity 



e At = {e Atlm )m _ 



(11.23) 



When s/m is under the hump while s lies beyond it (e.g., in Fig. 11.1 s — 4 
with m — 8), we can have 

\\e As \\2«\\e As/m \\^. (11.24) 

Unfortunately, rounding errors in the mth power of e As l m are usually small only 
relative to ||e Ax / m ||2, rather than \\e \\i. Thus, rounding errors may be a problem 
in using (11.23) to compute e Al . 

The Taylor series expansion in (11.8) is not a good method for computing e At . 
The reader should recall the example of catastrophic convergence in the compu- 
tation of e x (x e R) from Chapter 3 (Appendix 3.C). It is not difficult to imagine 
that the problem of catastrophic convergence in (11.8) is likely to be much worse, 
and much harder to contain. Indeed this is the case as shown by an example in 
Moler and Van Loan [7]. 

It was suggested earlier in this section that the series in (11.8) can be shown to 
converge by considering the diagonalization of A (assuming that A is nondefective). 
Suppose that A e C" x ", and that A possesses eigenvalues that are all distinct, and 
so we may apply (11.6). Since 



A k = [PAP -1 ]* 

= [PAP" 1 ][PAP" 1 ]---[PAP" 1 ][PAP _1 ] = PA k P~\ 
k factors 



(11.25) 
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we have 



-At 



y-A k t k = y-PA k p- 1 t k = p 



k=0 



k=0 



E-^ 



.k=Q 



k t k 



(oo 1 oo 1 OO . \ 

£=0 ' £=0 ' k=0 ' / 



= Pdmg(e Ao ',e 



Xi)t X\t 



')P~ 



If we define 



.At 



diag(e A 



J-\t 



"), 



(11.26) 



then clearly we can say that 



„Al 



= Y-A k t k = Pe M p- 



/t=0 



A:! 



(11.27) 



We know from the theory of Maclaurin series that e x — JZ'kLo H 1 ' conver g es f° r 
all x e R. Thus, e At converges for all t e R and is well defined, and hence the 
series in (11.8) converges, and so e At is well defined. Of course, all of this suggests 
that e may be numerically computed using (11.27). From Chapter 3 we infer that 
accurate, reliable means to compute e x (x is a scalar) do exist. Also, reliable 
methods exist to find the elements of A, and this will be considered later. But P 
may be close to singular, that is, the condition number k(P) (recall Chapter 4) may 
be large, and so accurate determination of P~ l , which is required by (11.27), may 
be difficult. Additionally, the approach (11.27) lacks generality since it won't work 
unless A is nondefective (i.e., can be diagonalized). Matrix factorization methods 
to compute e At (including those in the defective case) are considered in greater 
detail in Ref. 7, and this matter will not be mentioned further here. 

In Chapter 4 a condition number k(A) was defined that informed us about the 
sensitivity of the solution to Ax — b due to perturbations in A and b. It is possible 
to develop a similar notion for the problem of computing e At . From Golub and 
Van Loan [4], the matrix exponential condition number is 



v(A, t) — max 

I|£||2<1 



f e A ^-^Ee As 
Jo 



ds 



\\A\\ 2 
\\e At \\ 2 



(11.28) 



(The theory behind this originally appeared in Van Loan [12].) In this expression 
E e R" x " is a perturbation matrix. The condition number (11.28) measures the 



sensitivity of mapping A 
E such that 



„At 



for a given t e R. For a given t, there is a matrix 



AA+E)t 



„At 



112 



,At\ 



v(A,t) 



\E\\2 

\\A\\ 2 ' 



(11.29) 
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We see from this that if v(A, t) is large, then a small change in A (modeled by the 
perturbation matrix E) can cause a large change in e At . In general, it is not easy 
to specify A leading to large values for v(A, t). However, it is known that 

v(A,t)>t\\A\\ 2 (11.30) 

for t e R + with equality iff A is normal. (Any A e C" x " is normal iff A H A — 
AA H .) Thus, it appears that normal matrices are generally the least troublesome 
with respect to computing e At . From the definition of a normal matrix we see that 
real-valued, symmetric matrices are an important special case. 

Of the less dubious means to compute e At , Golub and Van Loan's algorithm 
11.3.1 [4] is suggested. It is based on Pade approximation, which is the use of 
rational functions to approximate other functions. However, we will only refer the 
reader to Ref. 4 (or Ref. 7) for the relevant details. A version of Algorithm 11.3.1 
[4] is implemented in the MATLAB expm function, and MATLAB provides other 
algorithm implementations for computing e At . 

11.4 THE POWER METHODS 

In this section we consider a simple approach to determine the eigenvalues and 
eigenvectors of A € R" x ". The approach is iterative. The main result is as follows. 

Theorem 11.11: Let A e R" x " be such that 

(a) A has n linearly independent eigenvectors x^\ corresponding to the eigen- 
Vahs{(X k ,xW)\k€Z n }. 

(b) The eigenvalues satisfy A,,_i € R with 

|A„_i| > \X n - 2 \ > \K-3\ >---|A,i| > | A. | 
(!„_! is the dominant eigenvalue). 
If yo € R" is a starting vector such that 

71-1 

;=o 
with a n -\ ^ 0, then for yk+i — Ayt with k e Z + 



lim ^Z£ = CX («-D (H.31) 

. k ■ ' 



for some c ^ 0, and (recalling that (x, y) — x T y — y T x for any x, y e R") 

(yn, A k yo) 
lim UU ' ,_, =A.„_i. (11.32) 

k^oo (y , A K 'yo) 
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Proof We observe that v* = A k yo, and since A k x < -^ — k-x^, we have 



n-2 



A k y Q = Y J ^.i^ U) +an-^ k n . l x in - V) , 



implying that 



A k y 



— Cln—lX 



(1-1) 



"•n-1 



n-2 

J2 a J 

.7=0 



^•B-l 



'' » *^. 



(11.33) 



Since |A./| < |A„_i| for _/ = 0, 1, . . . , n — 2, we have Unh- 
and hence 

A k yo 



\K-i) 



lim 



®n—\X 



(B-l) 



B-l 



Now A^o = jy i jZlajX k j x u \ so that 



B-l 



;=o 



and if 77 y = (yo, x^), then 



(yo, A^o> a n - X X k n _ x r] n -x + E" = o«/^1.7 



(JO, A*-iyo> a n -iX k n -_\r>n-i + JTjZUj^nj 



Vi-l 



k 1 



2 n -irin-l + 22j=o (l J r IJ [t— 



Wj-1 



fc-1 



Again, since lim,t_ 



(&)' 



(for j = 0, 1, . . . , n — 2), we have 



(yo, A*;yo> 

, llm 7 Ak-l \ = X n-i- 

k^oo (y , A K l y$) 



We note that 



1 > 



X n -2 


> 


^M-3 


A.B-1 




^n-1 



Ao 



Wi-1 



so the rate of convergence of A k yo/k k i _ l to a n _ix^" ^ is, according to (11.33), 
dominated by the term containing k n -ilk n -\. This is sometimes expressed 
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by writing 



A k y 



k n-\ 



-IX 



(n-1) 




= ^ . (11.34) 



The choice of norm in (11.34) is arbitrary. Of course, a small value for |A. n _2/A.„_i| 
implies faster convergence. 

The theory of Theorem 11.11 assumes that A is nondefective. If the algorithm 
suggested by this theorem is applied to a defective A, it will attempt to converge. 
In effect, for any defective matrix A there is a nondefective matrix close to it, and 
so the limiting values in (11.31) and (11.32) will be for a "nearby" nondefective 
matrix. However, convergence can be very slow, particularly if the dependent 
eigenvectors of A correspond to \ n -\, and A.„_2. In this situation the results may 
not be meaningful. 

If yo is chosen such that a„_i =0, rounding errors in computing y k +\ — Ay^ 
will usually give a y k with a component in the direction of the eigenvector x ( - n ~ l \ 
Thus, convergence ultimately is the result. To ensure that this happens, it is best 
to select yo with noninteger components. 

From (11.31), if A: is big enough, then 

y k = A k y Q * ck k n _ lX in - v \ 

and so y^ is an approximation to x'" -1 ^ (to within some scale factor). However, 
if |A.„_i| > 1, we see that \\yt\\ — > oo with increasing k, while if |l„_i| < 1, we 
see that \\yk\\ -> with increasing k. Either way, serious numerical problems will 
certainly result (overflow in the former case and rounding errors or underflow in the 
latter case). This difficulty may be eliminated by the proper scaling of the iterates, 
and leads to what is sometimes called the scaled power algorithm: 

Input N; { upper limit on the number of iterations } 
Input yo; { starting vector } 
y :=yo/l|yrjll2; { normalize yo to unit norm } 
k := 0; 

while k < N do begin 
z k+ t :=Ay k ; 

Yk+1 :=z /c+l/ll z /<+lll2; 
^+D:=y[ +1 ^ +1 ; 
k :=/f+1; 
end; 

In this algorithm yk+\ is the (k + l)th estimate of x < - n+l \ while A^ 1 ) is the 
corresponding estimate of eigenvalue \ n -\. From the pseudocode above we may 
easily see that 

yk = — r^— (11-35) 

l|A*y ||2 



TLFeBOOK 



THE POWER METHODS 501 

for all k > 1. With y Q = J2'jZo a j xU) («/ e R such that ||y || 2 = 1) 



n-2 



n-2 



A k y = Y j a j A k x u) + a n -\ A k x {n ~ l) = a^A^x 



(»-i) 



7=0 



7=0 



**<'> 



= a„-ik 



An-l) 



n-2 

■E 

7=0 



fl n-l \A. B — 1 



,0') 



•/(« 
,(*)! 



and as in Theorem 11.11 we see lim^oo ||m ( 'lb = 0, and (11.35) becomes 

(11.36) 



yk 



a n -ik k n _ 1 [x« , -» + u<V] 



\\an-i>£_i[x<- n -» + uW]\\ 2 



l^k 



x (n-\) + u (k) 
|| X («-D + M W|| 2 ' 



where fi^ is the sign of a n -\\ k _ x (i.e., ji^ e {+1, —1}). Clearly, as k — > oo, vector 
;y& in (11.36) becomes a better and better approximation to eigenvector x^" -1 '. To 
confirm that A^ +1 ^ estimates X n -i, consider that from (11.36) we have (for k 
sufficiently large) 



X (k+1) = yI +l Ay k+1 



( X (n-l)y Ax (n-1) 



|X(»-D||2 



X n -1 



[recall that Ax^" 1 ' = A.„_ix ( " _1) ]. 

If |A. n _i| = |A,„_2| > |A.;| for j = 0, 1, . . . , n — 3, we have two dominant eigen- 
values. In this situation, as noted in Quarteroni et al. [13], convergence may or may 
not occur. If, for example, k n -\ = X n -2, then vector sequence (y%) converges to 
a vector in the subspace of R" spanned by x^" -1 ', and x^ n ~ 2 \ In this case, since 
A e R nx '\ we must have A.„_i, X n -i e R, and hence for k > 1, we must have 

n-3 

A k y = J2 a J xl J xU) +««-2^_ 2 x ( "- 2) +«„_ 1 ^_ 1 x("- 1 \ 

7=0 



implying that 



A*yo 



"n-l 



— ctfi — \jv ~\~ (-ifi — 2" 



«-3 / i . \k 



so that 



lim 



A k y 



k^oo X 



— Clfj — ^ Jt ~\~ Clyi — 2 5 



n-1 
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which is a vector in a two-dimensional subspace of R". On the other hand, 



recall Example 11.3, where n — 2 so that Xq — e j6 \ X\ — e 



A* From (11.35) 



llwlb = 1, so, because A is a rotation operator, yt — A k yol\\A k yo\\2 will always 
be a point on the unit circle {(x, y)\x 2 + y 2 — 1}. Convergence does not occur 
since it is generally not the same point from one iteration to the next (e.g., consider 
— it/2 radians). 



Example 11.5 Consider an example based on application of the scaled power 
algorithm to the matrix 

"410 

A - 14 1 

1 4 



This matrix turns out to have the eigenvalues 

A = 2.58578644, k { = 4.00000000, A 2 = 5.41421356 

(as may be determined in MATLAB using the eig function). If we define y^ — 
[yic,o yic,\ yk,2\ T £ R 3 , then, from the algorithm, we obtain the iterates: 



k 


ykfi 


yk,\ 


yk,2 


*(*> 





0.57735027 


0.57735027 


0.57735027 


— 


1 


0.53916387 


0.64699664 


0.53916387 


5.39534884 


2 


0.51916999 


0.67891460 


0.51916999 


5.40988836 


3 


0.50925630 


0.69376945 


0.50925630 


5.41322584 


4 


0.50444312 


0.70076692 


0.50444312 


5.41398821 


5 


0.50212703 


0.70408585 


0.50212703 


5.41416216 


6 


0.50101700 


0.70566560 


0.50101700 


5.41420184 


7 


0.50048597 


0.70641885 


0.50048597 


5.41421089 


8 


0.50023215 


0.70677831 


0.50023215 


5.41421295 


9 


0.50011089 


0.70694993 


0.50011089 


5.41421342 


10 


0.50005296 


0.70703187 


0.50005296 


5.41421353 


11 


0.50002530 


0.70707101 


0.50002530 


5.41421356 



In only 1 1 iterations the power algorithm has obtained the dominant eigenvalue 
to an accuracy of eight decimal places. 

Continue to assume that A e R nxn is nondefective. Now define A M — A — jil 
(where / is the n x n identity matrix, as usual), and \i e R is called the shift 
(or shift parameter). 4 We will assume that /x always results in the existence of 
A~ l . Because A is not defective, there will be a nonsingular matrix P such that 
P~ l AP — A — diag(Ao, A.i, . . . , A„_i) (recall the basic facts from Section 11.2 
that justify this). Consequently 



A IA = PAP" 1 - fil ^ P _1 A M P = A-/j,I, 



(11.37) 



The reason for introducing the shift parameter /x will be made clear a bit later. 
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and so A" 1 has the eigenvalues 

Yk=- , keZ n . (11.38) 

Ajfc - [A 

P is a similarity transformation that diagonalizes A M , giving A — \il, so the eigen- 
values of (A — /x/) _1 must be the eigenvalues of A" 1 as these are similar matrices. 
A modification of the previous scaled power algorithm is the following shifted 
inverse power algorithm: 

Input N\ { upper limit on the number of iterations } 

Input yg; { starting vector } 

y :=yo/l|yoll2; 1 normalize yo to unit norm } 

k:=Q; 

while k < N do begin 

A n z k+1 '-=yk- 

y/c+1 :=Zfr+i/l|Z/r+ill2; 



Y 



(fr+1) 



= y[ +1 ^y/c+i; 



/c:=/c + 1; 
end; 

Assume that the eigenvalues of A satisfy 

|A„_i| > |A„- 2 |>---> IA.il > |A |, (11.39) 

and also that /z = 0, so then \kk\ = l/|Xfc|, and (11.39) yields 

iKol > iKil > \Yi\ > •■■> lK«-2l > \Yn-i\- (11-40) 

We observe that in the shifted inverse power algorithm A^Zk+i = y\ is equivalent 
to Zk+\ — A~ l yk, and so A" 1 effectively replaces A in the statement Zk+i '■— Ayj. in 
the scaled power algorithm. This implies that the shifted inverse power algorithm 
produces vector sequence (y^) that converges to the eigenvector of A~ l — A -1 
corresponding to the eigenvalue yo (= 1 Ao)- Since Ax^ — Xqx^ implies that 
A _1 jt' ' = y-x^ — Yqx^K then for a sufficiently large k, the vector y\ will approx- 
imate x {SS) . The argument to verify this follows the proof of Theorem 11.11. 
Therefore, consider the starting vector 

n-\ 

yo = ao x (0) + J2 a i xU) > a o # (H-41) 

;=i 

(such that Hyolb = 1)- We see that we must have 

1 "~ l 1 

A- k y Q = ao^x^ +Y j a j —x (j) 

X o j= i x j 

ra-1 

= aoy ^ (0) +2^ajYjX U) > ( n - 42 ) 

7=1 
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and thus 



a —k n ~ 1 

a k yo ( o) 



Immediately, because of (11.40), we must have 



t°j(z) xU) 



Yo 



lim 

k—>oo 



A k y 



aox (0) . 



From (11.42), we obtain 



A k yo = uqYq 











x®> 


n-l 

+E- 


(- 


\ x^ 




=„(« 





(11.43) 



In much the same way as we arrived at (11.35), for the shifted inverse power 
algorithm, we must have for all k > 1 



yk = 



A n k yo 

WAT/yoWi' 



(11.44) 



From (11.43) for \i 



yk 



0, this equation becomes 

aoYo[x (0) +v (k) l 
\\aoy£[xM+vWl\\2 



,(0) 



.,(*) 



Vk 



\\x 



(0) 



>(*)l 



b 



(11.45) 



where v^ e { + 1 , — 1 } . From the pseudocode for the shifted inverse power algorithm, 
if k is large enough, via (11.45), we obtain 



Y ik+1) = yI +1 Ay k+l 



(*< >) r Aj<°> 

I|X(°)||2 



= *0- 



(11.46) 



Thus, j/(* +1 ) is an approximation to A.o. 

In summary, for a nondefective A e R" x " such that (11.39) holds, the shifted 
inverse power algorithm will generate a sequence of increasingly better approxi- 
mations to the eigenpair (Xq, x^), when we set fx — 0. 

We note that the scaled power algorithm needs 0(n 2 ) flops (recall the definition 
of flops from Chapter 4) at every iteration. This is due mainly to the matrix-vector 
product step Zk+i — Ayt- To solve the linear system A^Zk+i — yk requires 0(« 3 ) 
flops in general. To save on computations in the implementation of the shifted 
inverse power algorithm, it is often best to LU -decompose A^ (recall Section 4.5) 
only once: A fl — LU. At each iteration LUzk+i — yk mav be solved using forward 
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and backward substitution, which is efficient since this needs only 0(n 2 ) flops at 
every iteration. Even so, because of the need to compute the LU decomposition of 
A M , the shifted inverse power algorithm still needs 0(« 3 ) flops overall. It is thus 
intrinsically more computation-intensive than is the scaled power algorithm. 

However, it is the concept of a shift that makes the shifted inverse power algo- 
rithm attractive, at least in some circumstances. But before we consider the reason 
for introducing the shift parameter /x, the reader should view the following example. 

Example 11.6 Let us reconsider matrix A from Example 11.5. If we apply 
the shifted inverse power algorithm using /a = to this matrix, we obtain the 
following iterates: 



k 


ykfi 


yk,l 


yk,2 


y(k) 





0.57735027 


0.57735027 


0.57735027 





1 


0.63960215 


0.42640143 


0.63960215 


5.09090909 


2 


0.70014004 


0.14002801 


0.70014004 


4.39215686 


3 


0.69011108 


-0.21792981 


0.69011108 


3.39841689 


4 


0.62357865 


-0.47148630 


0.62357865 


2.82396484 


5 


0.56650154 


-0.59845803 


0.56650154 


2.64389042 


6 


0.53330904 


-0.65662999 


0.53330904 


2.59925317 


7 


0.51623508 


-0.68337595 


0.51623508 


2.58886945 


8 


0.50782504 


-0.69586455 


0.50782504 


2.58649025 


9 


0.50375306 


-0.70175901 


0.50375306 


2.58594700 


10 


0.50179601 


-0.70455768 


0.50179601 


2.58582306 


11 


0.50085857 


-0.70589049 


0.50085857 


2.58579479 


12 


0.50041023 


-0.70652615 


0.50041023 


2.58578834 


13 


0.50019597 


-0.70682953 


0.50019597 


2.58578687 


14 


0.50009360 


-0.70697438 


0.50009360 


2.58578654 


15 


0.50004471 


-0.70704355 


0.50004471 


2.58578646 


16 


0.50002135 


-0.70707658 


0.50002135 


2.58578644 



We see that the method converges to an estimate of Xo (smallest eigenvalue of 
A) that is accurate to eight decimal places in only 16 iterations. 

As an exercise, the reader should confirm that the vector y^ in the bottom row 
of the table above is an estimate of the eigenvector for Xo. The reader should do 
the same for Example 11.5. 



Recall again that A l is assumed to exist (so that jj, is not an eigenvalue of A). 
Observe that Ax^ — Xtx^, so that (A — (il)x^ — (Xj — ix)x^\ and therefore 



A" 1 **/) 



Yj- 



C U) 



Suppose that there is an m e Z„ such that 



\X m - n\ < \Xj - fi\ 



(11.47) 



for all j e Z„ , but that j ^ m; that is, X m is closest to /x of all the eigenvalues of 
A. This really says that X m has a multiplicity of one (i.e., is simple). Now consider 
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the starting vector 



71-1 



>'o 



J2 a i x 



(./) 



Am) 



(11.48) 



;'=o 

jjtm 



with a m ^ 0, and \\y0W2 — 1- Clearly 



n-1 





V?o 


= \ 


(3 


jyfx (j) +a m yj, 






implying that 






- fl '« X + 2^ fl 7 ( 


Now via (11.47) 








XL 

Ym 


= 


Xj-11 


< 1. 


Therefore, via (11.49) 






lim 


A ~» 


k yo 





k (m) 



M) 



(11.49) 



k^oo Ym 

This implies that the vector sequence (yjt) in the shifted inverse power algorithm 
converges to x (pt \ Put simply, by the proper selection of the shift parameter /x, we 
can extract just about any eigenpair of A that we wish to (as long as X m is simple). 
Thus, in this sense, the shifted inverse power algorithm is more general than the 
scaled power algorithm. The following example illustrates another important point. 

Example 11.7 Once again we apply the shifted inverse power algorithm to 
matrix A from Example 11.5. However, now we select ji — 2. The resulting 
sequence of iterates for this case is as follows: 



k 


yk,o 


y*,i 


yk,2 


y(k) 





0.57735027 


0.57735027 


0.57735027 


— 


1 


0.70710678 


0.00000000 


0.70710678 


4.00000000 


2 


0.57735027 


-0.57735027 


0.57735027 


2.66666667 


3 


0.51449576 


-0.68599434 


0.51449576 


2.58823529 


4 


0.50251891 


-0.70352647 


0.50251891 


2.58585859 


5 


0.50043309 


-0.70649377 


0.50043309 


2.58578856 


6 


0.50007433 


-0.70700164 


0.50007433 


2.58578650 


7 


0.50001275 


-0.70708874 


0.50001275 


2.58578644 
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We see that convergence to the smallest eigenvalue of A has now occurred in 
only seven iterations, which is faster than the case considered in Example 1 1.6 (for 
which we used /i — 0). 

We see that this example illustrates the fact that a properly chosen shift param- 
eter can greatly accelerate the convergence of iterative eigenproblem solvers. This 
notion of shifting to improve convergence rates is also important in practical imple- 
mentations of QR iteration methods for solving eigenproblems (next section). 

So far our methods extract only one eigenvalue from A at a time. One may 
apply a method called deflation to extract all the eigenvalues of A under certain 
conditions. Begin by noting the following elementary result. 

Lemma 11.1: Suppose that B e R ( "" l)x( " _1) , and that B~ l exists, and that 
r e R" _1 , then 

1 r T I" 1 _ [ 1 -r T B~ x 
OB OB' 1 



(11.50) 



Proof Exercise. 

The deflation procedure is based on the following theorem. 

Theorem 11.12: Deflation Suppose that A n e R" x ", that eigenvalue A.,- € R 
for all i e Z„, and that all the eigenvalues of A n are distinct. The dominant eigenpair 
of A n is (A.„_i, x^" -1 '), and we assume that ||x^ _1 '||2 = 1. Suppose that Q n e 
R" x " is an orthogonal matrix such that Q„x (n_1) = [10 • • • 0] T — eo; then 



QnA n Q T n 



A-n-l a n -\ 

A„_i 



(11.51) 



Proof Q n exists because it can be a Householder transformation matrix (recall 
Section 4.6). Any eigenvector x^ of A n can always be normalized so that 

||*<*>|| 2 =1. 

Following (11.5), we have 



„[*C- 1 V' , - 2) ---;c (1 V 0) ] 










=T„ 












K-i 





• 










^«-2 • 


• 





.[.("-'V- 2 '...^ 1 '/ '] 










= T„ 








• ^! 













• 


Ao _ 



= D„ 
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that is, A n T n — T n D n . Thus, {Q T n — Q~ l via orthogonality) 



Now 



Q n A n Q T n = Q„T n D n T n - l Ql = {Q n T n )D n (Q n T n T x . 



QtJn = [eo Q n X 



(n-2) 



and via Lemma 11.1, we have 



QnX^QnX^] 



B, 



(11.52) 



1 *J_, 



n-l 



1 Cl 

B„-i 



Thus, (11.52) becomes 



G«A„G^ = 



1 b 1 

n- 

o B„- 



fi"- 1 ! 



X„-i o 
D n -i 



1 -fcl ,5 _1 



B 



n—\ n—\ 

1 



n-l 



^n-1 





*„-i( D »- 

B„ 



l — Ki-\h-\)B 



n-l 



■A-iV, 



which has the form given in (11.51). 

From Theorem 11.6, Q n A n Q^ and A„ are similar matrices, and so have the 
same eigenvalues. Via (11.51), A n -\ has the same eigenvalues as A n , except 
for k n -\. Clearly, the scaled power method could be used to find the eigenpair 
(X n -\, x*" -1 )). The Householder procedure from Section 4.6 gives Q n . From The- 
orem 11.12 we obtain A n -\, and the deflation procedure may be repeated to find 
all the remaining eigenvalues of A — A n . 

It is important to note that the deflation procedure may be improved with respect 
to computational efficiency by employing instead the Rayleigh quotient iteration 
method. This replaces the power methods we have considered so far. This approach 
is suggested and considered in detail in Golub and Van Loan [4] and Epperson [14]; 
we omit the details here. 



11.5 QR ITERATIONS 

The power methods of Section 11.4 and variations thereof such as Rayleigh quotient 
iterations are deficient in that they are not computationally efficient methods for 
computing all possible eigenpairs. The power methods are really at their best when 
we seek only a few eigenpairs (usually corresponding to either the smallest or 
the largest eigenvalues). In Section 11.4 the power methods were applied only to 
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computing real-valued eigenvalues, but it is noteworthy that power methods can 
be adapted to finding complex-conjugate eigenvalue pairs [19]. 

The QR iterations algorithms are, according to Watkins [15], due originally to 
Francis [16] and Kublanovskaya [17]. The methodology involved in QR iterations 
is based, in turn, on earlier work of H. Rutishauser performed in the 1950s. The 
detailed theory and rationale for the QR iterations are not by any means straight- 
forward, and even the geometric arguments in Ref. 15 (based, in turn, on the work 
of Parlett and Poole [18]) are not easy to follow. However, for matrices A e C" x " 
that are dense (i.e., nonsparse; recall Section 4.7), that are not too large, and that 
are nondefective, the QR iterations are the best approach presently known for find- 
ing all possible eigenpairs of A. Indeed, the MATLAB eig function implements a 
modern version of the QR iteration methodology. 5 

Because of the highly involved nature of the QR iteration theory, we will only 
present a few of the main ideas here. Other than the references cited so far, the 
reader is referred to the literature [4,6,13,19] for more thorough discussions. Of 
course, these are not the only references available on this subject. 

Eigenvalue computations such as the QR iterations reduce large problems into 
smaller problems. Golub and Van Loan [4] present two lemmas that are involved 
in this reduction approach. Recall from Section 4.7 that s(A) denotes the set of all 
the eigenvalues of matrix A (and is also called the spectrum of A). 

Lemma 11.2: If A € C" x " is of the form 



Aoo 




Aoi 
An 



where A 00 eC pxp , 
s(A n ). 



A i e C px i, An e C qxq (q + p = «), then s(A) = s(A 00 ) U 



Proof Consider 



Ax = 



Aoo Aoi 
An 



X2 



= X 



X] 

X2 



(x\ e C p and xi e C q ). If x-i ^ 0, then A\\X2 — Xx2, and so we conclude that 
X e s(An). On the other hand, if X2 — 0, then Aoo*i = Xx\, so we must have 
X € s(Aoo). Thus, s(A) e s(Aoo) U s(A\\). Sets s(A) and ,s(Aoo) U s(An) have 
the same cardinality (i.e., the same number of elements), and so s(A) — s(Aqq) U 
s(A n ). 

5 If A e C" x ", then [V, D] = eig(A) such that 

A = VDV~ l , 

where D 6 C" x " is the diagonal matrix of eigenvalues of A and V e C nx " is the matrix whose columns 
are the corresponding eigenvectors. The eigenvectors in V are "normalized" so that each eigenvector 
has a 2-norm of unity. 



TLFeBOOK 



510 



NUMERICAL METHODS FOR EIGENPROBLEMS 



Essentially, Lemma 11.2 states that if A is block upper triangular, the eigen- 
values lie within the diagonal blocks. 

Lemma 11.3: If A € C" xn , B e C pxp , X e C nxp (with p <n) satisfy 

AX = XB, rank(X) = p, (11.53) 

then there is a unitary Q e C nxn (so Q~ l — Q H ) such that 



Q"AQ=T = 



Too 7b i 
T n 



(11.54) 



where T m € C pxp , T m e C px(r, - p \ T n e c (n - p)x(n - p \ and s(T q) = 
s(A)r\s(B). 

Proof The QR decomposition idea from Section 4.6 generalizes to any X e 
Qnxp w j m p < n anc i rank(X) = /?; that is, complex- valued Householder matrices 
are available. Thus, there is a unitary matrix Q eC* M such that 



X=Q 



R 




where R e C pxp . Substituting this into (11.53) yields 



R 




where 



Too Tbi 


' R ' 




Tio 7n 


_ 


- 


Q H AQ = 


Too 





(11.55) 



From (11.55) T\qR — 0, implying that Tio = [yielding (11.54)], and also TqqR — 
RB, implying that B — R~ 1 TqqR (R _1 exists because X is full-rank). Too and 
B are similar matrices so s(B) — s(Tqo). From Lemma 11.2 we have s(A) — 
s(Too) U s (Tn)- Thus, s(A) — s(B) U s(T\i). From basic properties regarding sets 
(distributive laws) 

s(T 00 ) n s(A) = s(T 00 ) n [s(fi) U s(T n )] 

= [s(T O0 )ns(B)]\J[s(T O0 )ris(T 11 )] 

= s(T OQ )n0, 

implying that s(Tqq) — s(Tqo) D s(A) — s(A) D s(B). This statement [s(Toq) = 
s(A) n s(B)] really says that the eigenvalues of B are a subset of those of A. 
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Recall that a subspace of vector space C" is a subset of C" that is also a vector 
space. Suppose that we have the vectors xq, . . . , x m -\ e C"; then we may define 
the spanning set as 

m— 1 

span(^o, . . . , x m -i) — 2, ajXj\aj e C 

.7=0 



(11.56) 



In particular, if S — span(jt), where x is an eigenvector of A, then 

y e S =$■ Ay e S, 

and so S is invariant for A, or invariant to the action of A. It is a subspace 
(eigenspace) of C" that is invariant to A. Lemmas 11.2 and 11.3 can be used to 
establish the following important decomposition theorem (Theorem 7.4.1 in Ref. 
4). We emphasize that it is for real-valued A only. 

Theorem 11.13: Real Schur Decomposition If A e R" x ", then there is an 
orthogonal matrix Q e R" x " such that 



Q T AQ 



RoO ^01 

R n 











Ro,m-\ 
R\,m-l 



R 



m— \,m— 1 



n, 



(11.57) 



where each Rjj is either 1 x 1 or 2 x 2. In the latter case /?,-_,- will have a complex- 
conjugate pair of eigenvalues. 

Proof The matrix A e R" x ", so det(17 — A) has real-valued coefficients, and 
so complex eigenvalues of A always occur in conjugate pairs (recall Section 11.2). 
Let k be the number of complex-conjugate eigenvalue pairs in s(A). We will 
employ mathematical induction on k. 

The theorem certainly holds for k — via Lemmas 11.2 and 11.3 since real- 
valued matrices are only a special case. Now we assume that k > 1 (i.e., A possesses 
at least one conjugate pair of eigenvalues). Suppose that an eigenvalue is X — 
a + jfJ e s(A) with /5 # 0. There must be vectors x, y e R" (with y ^ 0) such that 

A(x + jy) = (a + jP)(x + jy), 



or equivalently 



A[xy] = [xy] 



-p 1 a 



(11.58) 



Since fJ ^ 0, vectors x and y span a two-dimensional subspace of R" that is 
invariant to the action of A because of (11.58). From Lemma 11.3 there is an 
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orthogonal matrix U\ e R" x " such that 



UfAUi 



Too 7b i 
Zii 



where T m € R 2x2 , 7bi e R 2x ^ n ~ 2 \ T u e R("- 2 ^(»- 2 \ and s(T 00 ) = {X, 1*}. By 
induction there is another orthogonal matrix t/2 such that t/JlnL^ has the neces- 
sary structure. Equation (11.57) then follows by letting 



Q = Ui 



h 

u 2 



where I2 is the 2x2 identity matrix. Of course, this process may be repeated as 
often as needed. 

A method that reliably gives us the blocks Rij for all i in (1 1.57) therefore gives 
us all the eigenvalues of A since Rjj is only 1 x 1, or 2 x 2, making its eigenvalues 
easy to find in any case. The elements in the first subdiagonal of Q T AQ of (11.57) 
are not necessarily zero-valued (again because /?,- j might be 2 x 2), so we say that 
Q 1 AQ is upper quasi-triangular. 

Definition 11.5: Hessenberg Form Matrix A € C" x " is in Hessenberg form 
if at j — for all i, j such that i — j > 1, 

Technically, A in this definition is upper Hessenberg. Matrix A in Example 4.4 
is Hessenberg. All upper triangular matrices are Hessenberg. The quasi-triangular 
matrix Q T AQ in Theorem 11.13 is Hessenberg. 

A pseudocode for the basic QR iterations algorithm is 

Input N\ { Upper limit on the number of iterations } 

Input A e R nx "; { Matrix we want to eigendecompose } 

Hq := qIaQq; { Reduce A to Hessenberg form } 

k:=V, 

while k < N do begin 

Hj<_i := Q/<R^; { QR-decomposition step } 

H k :=R k Q k ; 

/f:=/c + 1; 

end; 



In this algorithm we emphasize that A is assumed to be real-valued. Generalization 
to the complex case is possible but omitted. The statement Hq — QqAQ q generally 
involves applying orthogonal transformation <2o € R nxn to reduce A to Hessenberg 
matrix Hq, although in principle this is not necessary. However, there are major 
advantages (discussed below) to reducing A to Hessenberg form as a first step. 
The basis for this initial reduction step is the following theorem, which proves that 
such a step is always possible. 
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Theorem 11.14: Hessenberg Reduction If A € R" x ", there is an orthogonal 
matrix Q e R nx " such that Q T AQ — H is Hessenberg. 

Proof From Section 4.6 in general there is an orthogonal matrix P such that 
Px — ||x||2eo (e.g., P is a Householder matrix), where eo = [1 • • • 0] T e R" 
if x e R". 

Partition A according to 



A<°> = A = 



(0) 

'on 



«o 



tf 



where a^' € R and a , b e R"" 1 , A™ e r("-Dx(«-d_ Let P] be ormog onal such 
that P\ao — Hflolb^o ( e o e R" _1 ). Define 



Gi = 



1 o 
Pi 



1 y 

and clearly Q i — Q x (i.e., Q\ is also orthogonal). Thus 



id) 



d a<°>q[ 



(0) 

'in, 



b T P T 



Il«0ll2«0 Pl^W 



The first column of A^ satisfies the Hessenberg condition since it is 

[aL, 1 1 «o 1 1 2 0---0 ] T . The process may be repeated again by partitioning A^ 

n-2 zeros 
according to 

4(D uT 



id) 



1 oo 



'1 



[0 oi] A™ 



where A$ € R 2x2 , and [0 ai],b\ e R("" 2 ) x2 , A^ e R(»-2)x(n-2)_ Let p 2 be 
orthogonal such that P2«i = llailben ( e £ R" -2 )- Define 

G2 = P 2 



where 7 2 is the 2x2 identity matrix. Thus 



A (2) = G2A (i) e r = G2eiA (0) G[e r 



'1 ^2 



1 00 



°1 r 2 



[0 Hflilbeo] P 2 A ( 1 1 1 ) P 2 r 
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and the first two columns of A® satisfy the Hessenberg condition. Of course, we 
may continue in this fashion, finally yielding 

A ( "- 2) = Q n -2 • • • QiQiAQjQl ■ ■ ■ Q T n _ 2 , 

which is Hessenberg. We may define Q T = Q n -2- • • Q2Q1 an d H — A^" -2 ', 
which is the claim made in the theorem statement. 

Thus, Theorem 11.14 contains a prescription for finding Ho = QqAqQq as well 
as a simple proof of existence of the decomposition. Hessenberg reduction is done 
to facilitate reducing the amount of computation per iteration. Clearly, A and Hq 
are similar matrices and so possess the same eigenvalues. 

From the pseudocode for k — 1, 2, . . . , N, we obtain 

ffk-i = QkRt, 
H k = R k Q k , 



which yields 



H n = Q t n -QIq!h Q 1 Q 2 -Q n , (11.59) 



so therefore 

H N = Q T N -Q T 2 Q T l QlAQ Q l Q 2 ---Q N . (11.60) 

Matrices Hn and A are similar, and so have the same eigenvalues for any N . It is 
important to note that if Q k is constructed properly then H k is Hessenberg for all 
k. As explained in Golub and Van Loan [4, Section 7.4.2], the use of orthogonal 
matrices Q k based on the 2x2 rotation operator (matrix A from Example 11.3) 
is recommended. These orthogonal matrices are called Givens matrices, or Givens 
rotations. The result is an algorithm that needs only 0(n 2 ) flops per iteration 
instead of 0(n 3 ) flops. Overall computational complexity is still 0(n 3 ) flops, due 
to the initial Hessenberg reduction step. It is to be noted that the rounding error 
performance of the suggested algorithm is quite good [19]. 

We have already noted the desirability of the real Schur decomposition of A 
into 11 according to Q T AQ = 1Z in (1 1.57). In fact, with proper attention to details 
(many of which cannot be considered here), the QR iterations method is an excellent 
means to find Q and 1Z as in any valid matrix norm 

lim H N =11 (11.61) 

and 

Y\Qi = Q (11-62) 

1=0 
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(of course, Y\i=o Qi = Q0Q1 • • • Qn\ the ordering of factors in the product is 
important since Qi is a matrix for all j). The formal proof of this is rather difficult, 
and so it is omitted. 
Suppose that we have 









• 


• 


Q n 


1 





• 


• 


—a n -\ 





1 


• 


• 


—a„-2 








• 


• 


—(12 








• 


• 1 


—a\ 



€R" 



(11.63) 



It can be shown that 



p(X) = det(AJ - A) = X" + aiX n ~ l + a 2 X 



\ n—2 



a n -\X + a n . (11.64) 



Matrix A is called a companion matrix. We see that it is easy to obtain (1 1.64) from 
(11.63), or vice versa. We also see that A is Hessenberg. Because of (11.61), we 
may conceivably input A of (1 1.63) into the basic QR iterations algorithm (omitting 
the initial Hessenberg reduction step), and so determine the roots of p(X) = as 
these are the eigenvalues of A. Since p(X) is essentially arbitrary, except that it 
should not yield a defective A, we have an algorithm to solve the polynomial zero- 
finding problem that was mentioned in Chapter 7. Unfortunately, it has been noted 
[4,19] that this is not necessarily a stable method for finding polynomial zeros. 

Example 11.8 Suppose that 

p(X) = (X 2 -2X + 2)(X 2 - 4lX + 1) 

= X 4 - (2 + V2)X 3 + (2V2 + 3)X 2 - 2(1 + V2)X + 2, 



which has zeros for 



'-'^T^TT 



After 50 iterations of the basic QR iterations algorithm, we obtain 



#50 = 



0.5000 
0.2071 
0.0000 
0.0000 



-6.0355 
1.5000 
0.0000 
0.0000 



0.9239 

-0.3827 

0.7071 

0.7071 



5.3848 
-2.2304 
-0.7071 

0.7071 



^0,0 




-^0,1 
K1.1 



where R, ;j e R , The reader may wish to confirm that Rqq has the eigenvalues 
1 ± j, and that /?ij has the eigenvalues -4=(1 ± j). 
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On the other hand, for 

p(X) = (X + l)(X 2 -2X + 2)(X 2 - yflX + 1), 

which is a slight modification of the previous example, the basic QR iterations 
algorithm will fail to converge. 

The following point is also important. In (11.60), define Q — QqQi ■ ■ ■ Qn, 
so that H N = Q T AQ. Now suppose that A = A T . Clearly, Hj, = Q T A T Q = 
Q T AQ — Hn. This implies that Hn will be tridiagonal (defined in Section 6.5) 
for a real and symmetric A. Because of (11.61), we must now have 



lim H N = diag(fl ,o,.Ku, 

N— >oo 



■ , Rm-\,m-\) — D, 



where each /?, , is 1 x 1. Thus, D is the diagonal matrix of eigenvalues of A. Also, 
we must have YYhLo Qi as me corresponding matrix of eigenvectors of A. 

Example 11.9 Suppose that A is the matrix from Example 11.5: 



A = 



4 1 
1 4 1 
1 4 



We see that A is in Hessenberg form already. After 36 iterations of the basic QR 
iteration algorithm, we obtain 



#36 



5.4142 0.0000 0.0000 
0.0000 4.0000 0.0000 
0.0000 0.0000 2.5858 



so the matrix is diagonal and reveals all eigenvalues of A. Additionally, we have 



36 



UQi = 



0.5000 
0.7071 
0.5000 



-0.7071 
0.0000 
0.7071 



0.5000 

-0.7071 

0.5000 



which is a good approximation to the eigenvectors of A. 

We noted in Section 11.4 that shifting can be used to accelerate convergence in 
power methods. Similarly, shifting can be employed in QR iterations to achieve 
the same result. Indeed, all modern implementations of QR iterations incorporate 
some form of shifting for this reason. The previous basic QR iteration algorithm 
may be modified to incorporate the shift parameter ji e R. The overall structure of 
the result is described by the following pseudocode: 
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Input N; { Upper limit on the number of iterations } 

Input A g R" xn ; { Matrix we want to eigendecompose ] 

Ho := QqAQq; { Reduce A to Hessenberg form } 

k:=V, 

while k < A/do begin 

Determine the shift parameter /x e R; 

Hk--\ - fJ-l '■= QfcR/<; { QR-decomposition step } 

H k := R k Q k + nl; 

k:=k+\; 

end; 



The reader may readily confirm that we still have 



On 



QiQqAQoQi 



Qn 



just as we had in (11.60) for the basic QR iteration algorithm. Thus, we again find 
that Hk is similar to A for all k. Perhaps the simplest means to generate /i is the 
single-shift QR iterations algorithm: 

Input N\ { Upper limit on the number of iterations } 

Input A s R nxn ; { Matrix we want to eigendecompose } 

Hq := QqAQq; { Reduce >4 to Hessenberg form } 

k:=V, 

while k < N do begin 

Hk := H k _-\ (n - 1,n - 1); { \i k is the lower right corner element of H k _-\ } 

H/(_i - fi k l := Q k R k \ { QR-decomposition step } 

H k ■= RkQk + V; 

k:=k+-\; 

end; 

We note that /x is not fixed in general from one iteration to the next. Basically, \i 
varies from iteration to iteration in order to account for new information about s(A) 
as the subdiagonal entries of Hk converge to zero. We will avoid the technicalities 
involved in a full justification of this approach except to mention that it is flawed, 
and that more sophisticated shifting methods are needed for an acceptable algorithm 
(e.g., the double shift [4,19]). However, the following example shows that shifting 
in this way does speed convergence. 

Example 11.10 If we apply the single-shift QR iterations algorithm to A in 
Example 11.9, we obtain the following matrix in only one iteration: 



Hi = 



This matrix certainly does not have the structure of Hx, in Example 11.9, but the 
eigenvalues of the submatrix 



4.0000 


-1.4142 


0.0000 


1.4142 


4.0000 


0.0000 


0.0000 


0.0000 


4.0000 



4.0000 
-1.4142 



-1.4142 
4.0000 



are 5.4142, 2.5858. 
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Finally, we mention that our pseudocodes assume a user-specified number of 
iterations N. This is not convenient, and is inefficient in practice. Criteria to auto- 
matically terminate the QR iterations without user intervention are available, but a 
discussion of this matter is beyond our scope. 
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PROBLEMS 

11.1. Aided with at most a pocket calculator, find all the eigenvalues and eigen- 
vectors of the following matrices: 



(a) A = 



(b) B = 



(c) C 



4 
1 


1 

4 




" 





-2 " 


1 





1 





1 


2 


" 


1 


l - 

4 








1 

4 


1 





1 



(d) D 



\-j 

~j 2 

11.2. A conic section in R 2 is described in general by 



o!Xq + 2/3xoxi + Y x \ + & x o + e*i + p — 0. 



(ll.P.l) 



(a) Show that (ll.P.l) can be rewritten as a quadratic form: 

x T Ax + g T x + p = 0, (11.P.2) 

where x = [xq x\] t , and A — A T . 

(b) For a conic section in standard form A is diagonal. Suppose that A is 
diagonal. State the conditions on the diagonal elements that result in 
(11.P.2) describing an ellipse, a parabola, and a hyperbola. 

(c) Suppose that (11. P. 2) is not in standard form (i.e., A is not a diagonal 
matrix). Explain how similarity transformations might be used to place 
(11. P. 2) in standard form. 

11.3. Consider the companion matrix 



C = 









• 


• 


c n 


1 





• 


• 


— Cn-1 





1 


• 


• 


—C n -2 








• 


• 


-ci 








• 


• 1 


-c\ 



eC" 
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Prove that 

p n (X) = det(A7 - C) = X" + ciX"~ l + c 2 X n ~ 2 + ■■■ + c n -\X + c„, 

11.4. Suppose that the eigenvalues of A € C" x " are Xq, X\, . . . , X n -\. Find the 
eigenvalues of A + al , where I is the order n identity matrix and a e C 
is a constant. 

11.5. Suppose that A e R nxn is orthogonal (i.e., A -1 = A T ); then, if X is an 
eigenvalue of A, show that we must have \X\ — 1. 

11.6. Find all the eigenvalues of 

co&9 -sin0 

cos</> — sin</> 

sin0 cos« 

sin</> cos(/> 

(Hint: The problem is simplified by using permutation matrices.) 

11.7. Consider the following definition: A, B e C nxn are simultaneously diago- 
nalizable if there is a similarity matrix S e C" x " such that S~ l AS, and 
5 1-1 BS are both diagonal matrices. Show that if A, B e C" x " are simulta- 
neously diagonalizable, then they commute (i.e., Afi = BA). 

11.8. Prove the following theorem. Let A, B e C nxn be diagonalizable. There- 
fore, A and B commute iff they are simultaneously diagonalizable. 

11.9. Matrix A € C" x " is a square root of B e C nxn if A 2 = B. Show that every 
diagonalizable matrix in C" x " has a square root. 

11.10. Prove the following (Bauer-Fike) theorem (which says something about 
how perturbations of matrices affect their eigenvalues). If y is an eigenvalue 
of A + E € R nxn , and T~ l AT = D = diag(A , A-i, . . . , A.„_i), then 

min |A.-y|<K2(r)||E|| 2 . 

ie.«(A) 

Recall that s(A) denotes the set of eigenvalues of A (i.e., the spectrum of 
A). [Hint: If y e s(A), the result is certainly true, so we need consider 
only the situation where y <£ s(A). Confirm that if T~ 1 (A + E — yI)T 
is singular, then so is I + (D — yf)~ l (T~ l ET). Note that if for some 
B e R nx " the matrix I + B is singular, then (I + B)x — for some leR" 
that is nonzero, so ||x||2 = ||Bx||2, and so ||B||2 > 1- Consider upper and 
lower bounds on the norm ||(D — y I) -1 (T _1 ET)\\2-] 

11.11. The Daubechies 4-tap scaling function 4>(t) satisfies the two-scale difference 
equation 

<t>(t) = />o<M2f) + pi</>(2f - 1) + P2<P(2t - 2) + p 3 0(2r - 3), (11.P.3) 
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where supp</>(£) = [0,3] C R [i.e., 4>(t) is nonzero only on the interval 
[0, 3]], and where 

p =\(l + V3), pi = i(3 + V3), 

p 2 =5(3-V3), P3 = 5(l-V3). 

Note that the solution to (11. P. 3) is continuous (an important fact). 

(a) Find the matrix M such that 

M(/) = (/>, (11.P.4) 

where = [^(l)0(2)] r e R 2 . 

(b) Find the eigenvalues of M. Find the solution 4> to (11.P.4). 

(c) Take (p from (b) and multiply it by constant a such that a J2k=i W = 
1 (i.e., replace </> by the normalized form ouj)). 

(d) Using the normalized vector from (c) (i.e., a<j>), find 4>(k/2) for all 
keZ. 

[Comment: Computation of the Daubechies 4-tap scaling function is the 
first major step in computing the Daubechies 4-tap wavelet. The process 
suggested in (d) may be continued to compute <f>(k/2 J ) for any k e Z, and 
for any positive integer J. The algorithm suggested by this is often called 
the interpolatory graphical display algorithm (IGDA).] 

11.12. Prove the following theorem. Suppose A e R" x " and A — A T . Then A > 
iff A — P T P for some nonsingular matrix P e R" x ". 

11.13. Let A e R" x " be symmetric with eigenvalues 

^0 < ^1 < ' ' ' < ^n-2 < ^n-1- 

Show that for all x e R" 

Xqx x < x Ax < X n -ix x. 

[Hint: Use the fact that there is an orthogonal matrix P such that P T AP — 
A (diagonal matrix of eigenvalues of A). Partition P in terms of its row 
vectors.] 

11.14. Section 11.3 presented a method of computing e At "by hand." Use this 
method to 

(a) Derive (10.109) (in Example 10.9). 

(b) Derive (10.P.9) in Problem 10.23. 



TLFeBOOK 



522 



NUMERICAL METHODS FOR EIGENPROBLEMS 



(c) Find a closed-form expression for e , where 



A = 



k 1 
A 1 
A 



11.15. This exercise confirms that eigenvalues and singular values are definitely 
not the same thing. Consider the matrix 



A = 



2 

1 1 



Use the MATLAB eig and svd functions to find the eigenvalues and singular 
values of A. 

11.16. Show that e {A+B)t = e At e Bt for all t e R if AB = BA. Does e {A+B)t = 
e e Bt always hold for all t e R when AB ^ BA ? Justify your answer. 

11.17. This problem is an introduction to Floquet theory. Consider a linear system 
with state vector x(t) e R" for all t e R such that 



dx(t) 
dt 



A(t)x(t) 



(11.P.5) 



for some A(t) e R" x " (all t e R), and such that A(t + T) = A(t) for some 
T > [so that A(t) is periodic with period T]. Let <t>(f) be the fundamental 
matrix of the system such that 



d®(t) 



= A(f)0(f), cj>(0) = /. 



(a) Let *(f) = 0(t 



dt 

T), and show that 
dV(t) 



dt 



= A(0*(0- 



(11.P.6) 



(11.P.7) 



(b) Show that <D(f + T) = $(f)C, where C — $>(T). [Hint: Equations 
(11.P.6) and (11. P. 7) differ only in their initial conditions [i.e., ^(0) = 
what ?].] 

(c) Assume that C _1 exists for some C e R nx '\ and that there exists some 
R e R" x " such that C = e TR . Define P(t) = ®(t)e~ tR , and show that 
P(t + T) = P(t). [Thus, <D(f) = P(t)e tR , which is the general form of 
the solution to (11.P.6).] 

(Comment: Further details of the theory of solution of (11. P. 5) based on 
working with (11.P.6) may be found in E. A. Coddington and N. Levinson, 
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Theory of Ordinary Differential Equations, McGraw-Hill, New York, 1955. 
The main thing for the student to notice is that the theory involves matrix 
exponentials.) 



11.18. Consider the matrix 

A = 



4 2 
14 10 
14 1 
2 4 



(a) Create a MATLAB routine that implements the scaled power algorithm, 
and use your routine to find the largest eigenvalue of A. 

(b) Create a MATLAB routine that implements the shifted inverse power 
algorithm, and use your routine to find the smallest eigenvalue of A. 

11.19. For A in the previous problem, find K2(A) using the MATLAB routines that 
you created to solve the problem. 

11.20. If A € R nxn , and A — A T , then, for some x e R" such that x # 0, we 
define the Rayleigh quotient of A and x to be the ratio x T Ax/x T x (= 
(x, Ax) J ' {x, x)). The Rayleigh quotient iterative algorithm is described as 



/c:=0; 

while k < N do begin 

lx k :=z T k Az k lzlz k ; 

(A - n k l)y k := z k ; 

z/c+1 :=y/r/lly/cll2; 

k:=k+-\; 
end; 

The user inputs zo, which is the initial guess about the eigenvector. Note 
that the shift Hk is changed (i.e., updated) at every iteration. This has the 
effect of accelerating convergence (i.e., of reducing the number of iterations 
needed to achieve an accurate solution). However, A w = A — [x^l needs 
to be factored anew with every iteration as a result. Prove the following 
theorem. Let A e R nx " be symmetric and (k,x) be an eigenpair for A. If 
y «s x, ix — y T Ay/y T y with ||x|| 2 = | |y | b = 1, then 

\k-n\< ||A-A./|| 2 ||*-y||l. 

[Comment: The norm \\A — XI\\2 is an A-dependent constant, while \\x — 
y\\* = Hellj is the square of the size of the error between eigenvector x 
and the estimate y of it. So the size of the error between X and the estimate 
\i (i.e., \X — fi\) are proportional to IMI2 at the worst. This explains the 
fast convergence of the method (i.e., only a relatively small N is usually 
needed). Note that the proof uses (4.31).] 
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11.21. Write a MATLAB routine that implements the basic QR iteration algorithm. 
You may use the MATLAB function qr to perform QR factorizations. Test 
your program out on the following matrices: 



(a) 



4 2 
14 10 
14 1 
2 4 



(b) 



-\ 

1 ° ° 1 + 75 

1 -§->/2 

1 1 + V2 



Use other built-in MATLAB functions (e.g., roots or eig) to verify your 
answers. Iterate enough to obtain entries for Hn that are accurate to four 
decimal places. 

11.22. Repeat the previous problem using your own MATLAB implementation of 
the single-shift QR iteration algorithm. Compare the number of iterations 
needed to obtain four decimal places of accuracy with the result from the 
previous problem. 

11.23. Suppose that X,YeR nxn , and we define the matrices 



A = X + jY,B 



X 
Y 



-Y 
X 



Show that if A € s(A) is real-valued, then k e s(B). Find a relationship 
between the corresponding eigenvectors. 
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Numerical Solution of Partial 
Differential Equations 



12.1 INTRODUCTION 

The subject of partial differential equations (PDEs) with respect to the matter of 
their numerical solution is impossibly large to properly cover within a single chapter 
(or, for that matter, even within a single textbook). Furthermore, the development of 
numerical methods for PDEs is a highly active area of research, and so it continues 
to be a challenge to decide what is truly "fundamental" material to cover at an 
introductory level. In this chapter we shall place emphasis on wave propagation 
problems modeled by hyperbolic PDEs (defined in Section 12.2). We will consider 
especially the finite-difference time-domain (FDTD) method [8], as this appears to 
be gaining importance in such application areas as modeling of the scattering of 
electromagnetic waves from particles and objects and modeling of optoelectronic 
systems. We will only illustrate the method with respect to planar electromagnetic 
wave propagation problems at normal incidence. However, prior to this we shall 
give an overview of PDEs, including how they are classified into elliptic, parabolic, 
and hyperbolic types. 



12.2 A BRIEF OVERVIEW OF PARTIAL DIFFERENTIAL EQUATIONS 

In this section we define some notation and terminology that is used throughout 
the chapter. We explain how second-order PDEs are classified. We also summarize 
some problems that will not be covered within this book, simply citing references 
where the interested reader can find out more. 

We will consider only two-dimensional functions u(x, t) e R (or u(x, y) e R), 
where the independent variable x is interpreted as a space variable, and independent 
variable t is interpreted as time. The order of a PDE is the order of the highest 
derivative. For our purposes, we will never consider PDEs of an order greater than 
2. Common shorthand notation for partial derivatives includes 



du du d u d u 

, Uf ^ , Uxt — — , tlxx ~~ T • 

dx dt dtdx dx z 



An Introduction to Numerical Analysis for Electrical and Computer Engineers, by C.J. Zarowski 
ISBN 0-471-46737-5 © 2004 John Wiley & Sons, Inc. 
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If the PDE has solution w(x, y), where x and y are spatial variables, then often 
we are interested only in approximating the solution on a bounded region (i.e., a 
bounded subset of R 2 ). However, we will consider mainly a PDE with solution 
u(x,t), where, as already noted, t is time, and so we consider u(x,t) only for 
x e [a, b] = [xo, Xf] and t e [to, tf]. Commonly, to = 0, xo = 0, and x/ — L, with 
tf — T . We wish to approximate the PDE solution u(x, t) at grid points (mesh 
points) much as we did in the problem of numerically solving ODEs as considered 
in Chapter 10. Thus, we wish to approximate u(xk,t n ), where 

xi — xo + hk, t n = to + xn, (12.1) 

such that 

1 1 

h=— (xf-xo), T = —(t f -t ), (12.2) 

M N 

and so k = 0, 1, . . . , M, and n — 0, 1, . . . , N. This implies that we assume sam- 
pling on a uniform two-dimensional grid defined on the xt plane [(x, t) plane]. 
Commonly, the numerical approximation to u(xk, t n ) is denoted by Uk ,«• The index 
k is then a space index, and n is a time index. 

There is a classification scheme for second-order linear PDEs. According to 
Kreyszig [1], Myint-U and Debnath [2], and Courant and Hilbert [9], a PDE of 
the form 

Au xx + 2Bu xy + Cuyy — F (x , y, u, u x , u y ) (12.3) 

is elliptic if AC — B 2 > 0, parabolic if AC — B 2 — 0, and is hyperbolic if AC — 
B 2 < 0. 1 It is possible for A, B, and C to be functions of x and y, in which case 
(12.3) may be of a type (elliptic, parabolic, hyperbolic) that varies with x and y. For 
example, (12.3) might be hyperbolic in one region of R 2 but parabolic in another. 
Of course, the terminology as to type remains the same when space variable y is 
replaced by time variable t. 

An example of an elliptic PDE is the Poisson equation from electrostatics [10] 

V xx + Vyy = — p(x,y), (12.4) 

€ 

where the solution V(x, y) is the electrical potential (e.g., in units of volts) at the 
spatial location (x, y) in R 2 , constant e is the permittivity of the medium (e.g., 
in units of farads per meter), and p(x, y) is the charge density (e.g., in units of 
coulombs per square meter) at the spatial point (x, y). We have assumed that 
the permittivity is a constant, but it can vary spatially as well. Certainly, (12.4) 

This classification scheme is related to the classification of conic sections on the Cartesian plane. The 
general equation for such a conic on R is 

Ax 1 + Bxy + Cy 2 + Dx + Ey + F = 0. 

The conic is hyperbolic if B 2 - AAC > 0, parabolic if B 2 - 4 AC = 0, and is elliptic if B 2 - 4 AC < 0. 
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has the form of (12.3), where for u(x, y) — V(x, y) we have B — 0, A — C — 
1, and F(x, y, u, u x , u y ) — —p(x, y)/e. Therefore, AC — B 2 — 1 > 0, confirming 
that (12.4) is elliptic. 

To develop an approximate method of solving (12.4), one may employ finite 
differences. For example, from Taylor series theory 

d 2 V(x k , Jn) _ V(x k +u Jn) ~ 2V(x k , y n ) + V(x k -i, y„) h 2 9 4 Vfe, y n ) 

dx 2 h 2 12 dx 4 

(12.5a) 

for some %\ e [xk-i, Xk+i], and 

d 2 V(x k ,y„) _ Vjxk, y«+i) - 2V(x k , y n ) + V(xk, y«-i) _ r 2 d 4 V(x k , rj„) 

dy 2 ~ x 2 12 3y 4 

(12.5b) 

for some r\ n e [y n -\, y n +\\, where x k — xq + hk, y n — yo + xn [recall (12.1), and 

(12.2)]. The finite-difference approximation to (12.4) is thus 



Vk+\,n - 2Vk,n + Vk-hn Vk,n+\ ~ 2V k ,n + Vk,n-\ _ PJXk, yn) 

h 2 x 2 e 



(12.6) 



Here k = 0, 1, . . . , M, and n = 0, 1, . . . , N. Depending on p(x, y) and boundary 
conditions, it is possible to rewrite (12.6) as a linear system of equations in the 
unknown (approximate) potentials Vj. „. In practice, N and M may be large, and so 
the linear system of equations will be of high order consisting of O (NM) unknowns 
to solve for. Because of the structure of (12.6), the linear system is a sparse one, 
too. It has therefore been pointed out [3,7,11] that iterative solution methods are 
preferred, such as the Jacobi or Gauss-Seidel methods (recall Section 4.7). This 
avoids the problems inherent in storing and manipulating large dense matrices. 
Epperson [11] notes that in recent years conjugate gradient methods have begun to 
displace Gauss-Seidel/Jacobi approaches to solving large and sparse linear systems 
such as are generated from (12.6). In part this is due to the difficulties inherent 
in obtaining the optimal value for the relaxation parameter a> (recall the definition 
from Section 4.7). 

An example of a parabolic PDE is sometimes called the heat equation, or diffu- 
sion equation [2-4, 11] since it models one-dimensional diffusion processes such 
as the flow of heat through a metal bar. The general form of the basic parabolic 
PDE is 

u, — a 2 u xx (12.7) 

for x e [0, L], t e R + . Here u(x, t) could be the temperature of some material at 
(x,t). It could also be the concentration of some chemical substance that is diffusing 
out from some source. (Of course, other physical interpretations are possible.) 
Typical boundary conditions are 

u(0,t) = 0, u(L,t) = 0for t > 0, (12.8a) 
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and the initial condition is 

u(x,0) = f(x). (12.8b) 

The initial condition might be an initial temperature distribution, or chemical con- 
centration. If uix,t) is interpreted as temperature, then the boundary conditions 
(12.8a) state that the ends of the one-dimensional medium are held at a constant 
temperature of (e.g., degrees Celsius). 

Equation (12.7) has the form of (12.3) with B — C — 0, and hence AC - B 2 = 
0, which is the criterion for a parabolic PDE. Finite-difference schemes analogous 
to the case for elliptic PDEs may be developed. Classically, perhaps the most 
popular choice is the Crank-Nicolson method summarized by Burden and Faires 
[3], but given a more detailed treatment by Epperson [11]. 

Another popular numerical solution technique for PDEs is the finite -element 
(FEL) method. It applies to a broad class of PDEs, and there are many commercially 
available software packages that implement this approach for various applications 
such as structural vibration analysis, or electromagnetics. However, we will not 
consider the FEL method as it deserves its own textbook. The interested reader can 
see Strang and Fix [5] or Brenner and Scott [6] for details. A brief introduction 
appears in Burden and Faires [3]. 

As stated earlier, the emphasis in this book will be on wave propagation prob- 
lems as modeled by hyperbolic PDEs. We now turn our attention to this class 
of PDEs. 



12.3 APPLICATIONS OF HYPERBOLIC PDEs 

In this section we summarize two problems that illustrate how hyperbolic PDEs 
arise in practice. In later sections we will see that although both involve the mod- 
eling of waves propagating in physical systems, the numerical methods for their 
solution are different in the two cases, and yet they have in common the application 
of finite-difference schemes. 

12.3.1 The Vibrating String 

Consider an elastic string with its ends fixed at the points x — and x — L (so 
that the string is of length L unstretched). If the string is plucked at position 
x = xp ixp e (0, L)) at time t — such as shown in Fig. 12.1, then it will vibrate 
for t > 0. The PDE describing w(x, t), which is the displacement of the string at 
position x and time t, is given by 

u tt = c 2 u xx . (12.9) 

The system of Fig. 12.1 is also characterized by the boundary conditions 

u(0,t) = 0, u(L,t) = for all t e R + , (12.10) 
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Figure 12.1 An elastic string plucked at time t = at point P, which is located at x = xp. 

which specify that the string's ends are fixed, and we have the initial conditions 

du(x, t) 



u(x,0) = f(x), 



dt 



g(x), 



(12.11) 



r=0 



which describes the initial displacement, and velocity of the string, respectively. 
As explained, for example, in Kreyszig [1] or in Elmore and Heald [12], the PDE 
(12.9) is derived from elementary Newtonian mechanics based on the following 
assumptions: 

1. The mass of the string per unit of length is a constant. 

2. The string is perfectly elastic, offers no resistance to bending, and there is 
no friction. 

3. The tension in stretching the string before fixing its ends is large enough to 
neglect the action of gravity. 

4. The motion of the string is purely a vibration in the vertical plane (i.e., the 
y direction), and the deflection and slope are small in absolute value. 

We will omit the details of the derivation of (12.9), as this would carry us too far 
off course. However, note that constant c 2 in (12.9) is 



c 2 = 



T 

P ' 



(12.12) 



where T is the tension in the string (e.g., units of newtons) and p is the density of 
the string (e.g., units of kilograms per meter). A dimensional analysis of (12.12) 
quickly reveals that c has the units of speed (e.g, meters per second). It specifies 
the speed at which waves propagate on the string. 
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It is easy to confirm that (12.9) is a hyperbolic PDE since, on comparing (12.9) 
with (12.3), we have A = c 2 , B = 0, and C = -1. Thus, AC - B 2 = -c 2 < 0, 
which meets the definition of a hyperbolic PDE. 

At this point we summarize a standard method for obtaining series-based analyt- 
ical solutions to PDEs. This is the method of separation of variables (also called the 
product method). We shall also find that Fourier series expansions (recall Chapters 1 
and 3) have an important role to play in the solution method. The solutions we 
obtain yield test cases that we can use to gauge the accuracy of numerical methods 
that we consider later on. 

Assume that the solution 2 to (12.9) can be rewritten in the form 

u(x,t) = X(x)T(t). (12.13) 

Clearly 

u xx = X xx T, u tt =XT tl , (12.14) 

which may be substituted into (12.9), yielding 

XT tt = c 2 TX xx , (12.15) 

or equivalently 

lit Ayr 

-£- = -^L, (12.16) 

c 2 T X 

The expression on the left-hand side is a function of t only, while that on the 
right-hand side is a function only of x. Thus, both sides must equal some constant, 
say, k: 

±tt Arv 

-£- = ^- = k. (12.17) 

c 2 T X 

From (12.17) we obtain two second-order linear ODEs in constant coefficients 

X xx -kX = (12.18) 

and 

T tt -c 2 icT = 0. (12.19) 

We will now ascertain the general form of the solutions to (12.18) and (12.19), 
based on the conditions (12.10) and (12.11). From (12.10) substituted into (12.13), 
we obtain 

u(0, t) = X(0)T(t) = 0, u(L, t) = X(L)T(t) = 0. 

Theories about the existence and uniqueness of solutions to PDEs are often highly involved, and so 
we completely ignore this matter here. The reader is advised to consult books dedicated to PDEs and 
their solution for such information. 
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If T(t) — (all i), then u(x, t) — for all x and t. This is the trivial solution, and 
we reject it. Thus, we must have 

X(0) = X(L) = 0. (12.20) 

For k = 0, Eq. (12.18) is X xx — 0, which has the general solution X(x) — ax + b, 
but from (12.20) we conclude that a — b — 0, and so X(x) — for all x. This is the 
trivial solution and so is rejected. If k — /x 2 > 0, we have ODE X xx — y?X — 0, 
which has a characteristic equation possessing roots at ±/z. Consequently, X(x) — 
ae ,LX +be~ flx . If we apply (12.20) to this, we conclude that a = b = 0, once 
again giving the trivial solution X(x) — for all x. Now finally suppose that 
k — —fi 2 < 0, in which case (12.18) becomes 

X XX +/3 2 X = 0, (12.21) 

which has the characteristic equation .v 2 + 1 — 0. Thus, the general solution to 
(12.21) is of the form 

X(x) = a confix) + bsin(fix). (12.22) 

Applying (12.20) yields 

X(Q) = a = 0, X(L) = & sinOSL) = 0. (12.23) 

Clearly, to avoid encountering the trivial solution, we must assume that b ^ 0. 
Thus, we must have 

sinOSL) = 0, 

implying that we have 

nit 
P = — , n e Z. (12.24) 

However, we avoid fi — (for n — 0) to prevent X(x) — for all x; and we 
consider only n e {1, 2, 3, . . .} = N because sin(— x) — — sinx, and the minus sign 
can be absorbed into the constant b. Thus, in general 

X(x) = X„(x) = 6„sin(— x) (12.25) 

for « € N, and where x € [0, L]. So now we have found that 



in which case (12.19) takes on the form 



T„+A 2 r = 0, where X n = —. (12.26) 
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This has a general solution of the form 

T n (t) = A n cos(X„t) + B n sin(A.„f) (12.27) 

again for n e N. Consequently, u n {x, t) — X n (x)T n (t) is a solution to (12.9) for 
all n e N, and 

u n (x, t) = [A n cos(X n t) + B„ sm(X n t)] sin I — x 1 . (12.28) 

The functions (12.28) are eigenfunctions with eigenvalues X n — rnic/L for the PDE 
in (12.9). The set {X n \n e N} is the spectrum. Each u n (x,t) represents harmonic 
motion of the string with frequency X„/(2jt) cycles per unit of time, and is also 
called the nth normal mode for the string. The first mode for n = 1 is called the 
fundamental mode, and the others (for n > 1) are called overtones, or harmonics. 
It is clear that u n (x, t) in (12.28) satisfies PDE (12.9), and the boundary conditions 
(12.10). However, u n (x,t) by itself will not satisfy (12.9), (12.10) and (12.11) 
all simultaneously. In general, the complete solution is [using superposition as the 
PDE (12.9) is linear] 

u(x, t) = ^ u„(x, t) = ^[A„ cos(A„f) + B„ sm(X„t)] sin (^-*) , (12.29) 

n=\ n=\ 

where the initial conditions (12.11) are employed to find the series coefficients A n 
and B n for all n. We will now consider how this is done in general. 
From (12.11) we obtain 

m(x,0) = ^A^sin^— x\ = f(x) (12.30) 

M = l 



and 








di 


(x, 
dt 


"i 


=0 = 



2_][—A n X n sm(X n t) + B n X n cos(X n t )] sin I — x ) 

.« = ! 



= *(*) 

r=0 



oo 

E/H7T \ 
B n X n sm{—x)=g(x). (12.31) 

n=\ 

The orthogonality properties of sinusoids can be used to determine A n and B n 
for all n. Note that (12.30) and (12.31) are particular instances of Fourier series 
expansions. In particular, observe that for k, n e N 

i, Sm (lT X ) Sm (^j^ = { L/2, n = k ■ (12J2) 
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Plainly, set {sm(njr x / L)\n e N} is an orthogonal set. Thus, from (12.30) 
2 f L /kit \ ^-^ /rnt \\ 2 [ L /lor 



i ^(r) ^ A » sm (r)r I= i/o /(x)sin 



L 



x \ dx, 



and via (12.32) this reduces to 



2 f L (kit \ 

Ak = — I f{x) sin I — x I dx, 



(12.33) 



and similarly from (12.31), and (12.32) 



If 1 

^■kL Jo 



B k - — T I g(x) sin ( —x ) dx. 



(12.34) 



Example 12.1 Suppose that g(x) — for all x. Thus, the initial velocity of the 

string is zero. Let the initial position (deflection) of the plucked string be triangular 

such that 

2H L 
x, < x < — 

/U) { 2 L h l ^ . (12.35) 

(L — x), — < x < L 

L 2 



In Fig. 12.1 this corresponds to xp — L/2 and u(xp, 0) = H . Since g(x) — for 
all x via (12.34), we must have B^ — for all k. From (12.33) we have 



AH 



f L/2 ( k * \ ( 

J x sin I — x \ dx + I 
Jo \ L / J L 



L ( klT , 

(L — x) sin — x dx 

L/2 \L ' 



Since 



/ 



sin(flx) dx — cos(ax) + C 

a 



and 



f ! ! 

/ x sin(ax) dx — x cos(ax) H r- sin(ax) + C 

J a a 1 



(C is a constant of integration), on simplification we have 



SH (kit 



(12.36) 



TLFeBOOK 



534 



NUMERICAL SOLUTION OF PARTIAL DIFFERENTIAL EQUATIONS 



0.1 -s 




Time (/) 







Position (x) 



Figure 12.2 Fourier series solution to the vibrating string problem. A mesh plot of u(x, t) 
as given by Eq. (12.37) for the parameters L 
the first 100 terms of the series expansion. 



as given by Eq. (12.37) for the parameters L = 1, c/L = A, and H = -At. The plot employed 



Thus, substituting (12.36) into (12.29) yields the general solution for our example, 
namely 



u(x, t) — 



%H ^ 1 (kn\ (kite \ (kit 



but sin(&7r/2) = for even k, and so this expression reduces to 

8H ^ (-1)"" 1 ({2n-\)jtc\ /(2«-1)tt 
u(x, t) — — z- > ^ cos I 1 I sin I x 

n z / -~ i (2« — \y 



(12.37) 



Figure 12.2 shows a typical plot of the function u(x, t) as given by (12.37) (for 
the parameters stated in the figure caption). 

The reader is encouraged to think about Fig. 12.2, and to ask if the picture is 
a reasonable one on the basis of his/her intuitive understanding of how a plucked 
string (say, that on a stringed musical instrument) behaves. Of course, this question 
should be considered with respect to the modeling assumptions that lead to (12.9), 
and that were listed earlier. 

12.3.2 Plane Electromagnetic Waves 

An electromagnetic wave (e.g., radio wave or light) in three-dimensional space R 3 
within some material is described by the vector magnetic field intensity H(x, y, z, t) 
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[e.g., in units of amperes per meter, (A/m)] and the vector electric field intensity 
E(x, y, z, t) [e.g., in units of volts per meter (V/m)] such that 

H(x, y, z, t) — H x (x, y, z, t)x + H y (x, y, z, t)y + H z (x, y, z, t)z, 
E(x, y, z, t) — E x (x, y, z, t)x + E y (x, y, z, t)y + E z {x, y, z, t)z, 

where x, y, and z are the unit vectors in the x, y, and z directions of R 3 , respec- 
tively. The dynamic equations that H and E both satisfy are Maxwell's equations: 

— 95 

V x E = (Faraday's law), (12.38) 

dt 

— 3D 

V x H (Ampere's law). (12.39) 

dt 

Here the material in which the wave propagates contains no charges or cur- 
rent sources. The magnetic flux density B(x, y, z,t), and the electric flux density 
D(x, y, z,t) are assumed to satisfy 

D = €£, ~B = n~H. (12.40) 

These relations assume that the material is linear, isotropic [i.e., the same in all 
directions, and homogeneous i.e., the parameters € and /x do not vary with (x, y, z)]. 
Constant e is the material's permittivity [units of farads per meter (F/m)], and 
constant /x is the material's permeability [units of henries per meter (H/m)]. The 
permittivity and permeability of free space (i.e., a vacuum) are often denoted by 
eo and /xo, respectively, and 

6 = e r 6o, /"- = /"-i-Mo, (12.41) 

where e r is the relative permittivity and \x r is the relative permeability of the 
material. Note that 



eo 



8.854185 x 10" 12 F/m, p Q = 400tt x 10" 9 H/m. (12.42) 



If the material is air, then e r « 1 and \i r « 1 to very good approximation, and 
so air is not practically distinguished (usually) from free space. For commonly 
occurring dielectric materials (i.e., insulators), we have p r & 1, also to excellent 
approximation. On the other hand, for magnetic materials (e.g., iron, cobalt, nickel, 
various alloys and mixtures), fi r will be very different from unity, and in fact 
the relationship B — jiH must often be replaced by sometimes quite complicated 
nonlinear relationships, often involving the phenomenon of hysteresis. But we will 
completely avoid this situation here. 
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In general, for the vector field A — A x x + A y y + A z z, the curl is the determinant 



V x A 



X 


y 


z 


a 


3 


3 


dx 


By 


Bz 


A x 


Ay 


A z 



(12.43) 



so, expanding this expression with A — E and A — H gives (respectively) 



V x E = x 
and 



dE z 
~~By~ 



V x H 



— „ /3H : 



By 



BEy 

~~dz 



8Hy 

~Jz 



+ y 



dE r dE 7 



dz 



BHx 

dz 



dx 



dH z 

dx 



9Ey 

dx 



dH y 

dx 



BE, 

By 



BH X 

By 



(12.44) 



(12.45) 



We will consider only transverse electromagnetic (TEM) waves (i.e., plane 
waves). If such a wave propagates in the x direction, then we may assume that 
E x — E z — 0, and so H x — H y — 0. Note that the electric and magnetic field com- 
ponents E y and H z are orthogonal to each other. They lie within the (y, z) plane, 
which itself is orthogonal to the direction of travel of the plane wave. From (12.44) 
and (12.45), Maxwell's equations (12.38) and (12.39) reduce to 



BEy 



BH 7 



dH 7 



BE V 



— —I 1 

Bx Bt 



— —€ 

Bx Bt 



(12.46) 



where we have used (12.40). Combining the two equations in (12.46), we obtain 
either 

B 2 H 7 B 2 H 7 



— ixe- 



Bx 2 ^ Bt 2 ' 
which is the wave equation for the magnetic field, or 



(12.47) 



B l E v 



B l E v 



lie- 



dx 2 ' dt 2 ' 
which is the wave equation for the electric field. If we define 

1 



//xe 



(12.48) 



(12.49) 



then the general solution to (12.48) (for example) can be expressed in the form 

E y (x, t) = E Jr (x - vt) + Ey,(X + vt), (12.50) 
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where the first term is a wave propagating in the +x direction (i.e., to the right) 
with speed v and the second term is a wave propagating in the — x direction (i.e., 
to the left) with speed v? Equation (12.50) is the classical D'Alembert solution to 
the scalar wave equation (12.48). Clearly, similar reasoning applies to (12.47). Of 
course, using (12.49) in (12.48), we can write 



d z E 



dt 2 



y _ v 2 9 E y 



dx 2 



(12.51) 



which has the same form as (12.9). In short, the mathematics describing the vibra- 
tions of mechanical systems is much the same as that describing electromagnetic 
systems, only the physical interpretations differ. Of course, (12.51) clearly implies 
that (12.47) and (12.48) are hyperbolic PDEs. 



Example 12.2 It is easy to confirm that (12.37) can be rewritten in the form 
of (12.50). Via the identity 



cos A sinB = ±[sin(A + B) - sin(A - B)] 



(12.52) 



we see that 



cos 



(2n — \)nc 



(2m — \)tx 



1 
-- sin 

2 



(2m — 1)tt 
L 
(2m — 1)jt 
L 



(x + ct) 
(x — ct) 



Thus, (12.37) may immediately be rewritten as 



AH ~ (-If 1 . 

U(X, t) — =- > pr sin 

n 2 *-*• (2m - l) 2 

n=\ 



(2m - 1)71 

L 



(x — ct) 



=u\(x—ct) 



AH ^ (-1)" -1 
It 2 2-> (2m - l) 2 

n=\ 



sin 



(2m - Y)it 



(x + ct) 



=u i (x+ct) 

We note that when e r — /x r = 1, we have v — c, where 

1 



V^ofo 



(12.53) 



Readers are invited to draw a simple sketch and convince themselves that this interpretation is cor- 
rect. This interpretation is vital in understanding the propagation of electromagnetic waves in layered 
materials, such as thin optical films. 
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This is the speed of light in a vacuum. Since for real materials p, r > \ and e,- > 1, 
we have u < c, so an electromagnetic wave cannot travel at a speed exceeding that 
of light in a vacuum. 

Now we will assume sinusoidal solutions to the wave equations such as would 
originate from sinusoidal sources. 4 Specifically, let us assume that 

E y (x,t) — E sm(cot - fix), (12.54) 

where p — co^/JIe — 2jt/X, and X is the wavelength (e.g., in units of meters). The 
frequency of the source is a>, a fixed constant, and so the wavelength will vary 
depending on the medium. If the free-space wavelength is denoted A.rj, then 

2tt &) 

— = -, (12.55) 

where c is from (12.53). If the free-space wave then propagates into a denser 

material, then 

7.71 a> 

— = - (12.56) 

k v 

for v given by (12.49). From (12.55) and (12.56), we obtain 

k = -A. . (12.57) 

c 

Since v < c, we always have k < ko; that is, the wavelength will shorten. This 
observation is useful in checking numerical methods that model the propagation 
of sinusoidal waves across interfaces between different materials (e.g., layered 
structures such as thin optical films). 
From (12.54) we have 

dEy 

= —/3EQCOs(cLit — fix) 

dx 

so that from the first equation in (12.46) we have 



-J 



PEq fJE 

cos(cot — fix) dt — sin(ft)? — fix). (12.58) 



The characteristic impedance of the medium in which the sinusoidal electromag- 
netic wave travels is defined to be 

H z p V e V ^r 

For example, the Colpitts oscillator from Chapter 10 could operate as a sinusoidal signal generator to 
drive an antenna, thus producing a sinusoidal electromagnetic wave (radio wave) in space. 
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where Zq — V/^o7^0 ls the characteristic impedance of free space. The units of 
Z are in ohms (£2). We see that Z is analogous to the concept of impedance that 
arises in electric circuit analysis. 

The analogy between our present problem and phasor analysis in basic electric 
circuit theory can be exploited. For suitable E(x) e R 

E y = E y (x, t) = E(x)e ja " (12.60) 

so that 

d 2 E y d 2 E(x) ,„, 9 2 £ v , 

1^ = ^^ ' V = "^ W ' (12 ' 61) 



Substituting (12.61) into (12.48) yields 

e ja " = -/xew z £'(x)e 



d2E <- X K icot _ _„ £ ,.,2 ,,,,.„>,/ 



JX 2 

which reduces to the second-order linear ODE 

d 2 E(x) , 

V^ + M«« 2 £(x) = 0. (12.62) 

For convenience, we define the propagation constant 

y = jco^Jie = jf3, (12.63) 

so — y 2 — peco 2 , and (12.62) is now 

d 2 E 7 

— - y 2 E = (12.64) 

(E — E(x)). This ODE has a general solution of the form 

E(x) = E e- yx + E ie YX , (12.65) 

where Eq and E\ are constants. Recalling (12.60), it follows that 

E y (x, t) = £oe ; ^ _yx + Eie jmt+yx 

= Eoe-JV x - M) + Eie i{fix+mt) . (12.66) 

Of course, the first term is a wave propagating to the right, and the second term is 
a wave propagating to the left. 

So far we have assumed wave propagation in lossless materials since this is 
the easiest case to consider at the outset. We shall now consider the effects of 
lossy materials on propagation. This will be important in that it is a more realistic 
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assumption in practice, and it is important in designing perfectly matched layers 
(PMLs) in the finite-difference time-domain (FDTD) method, as will be seen later. 
We may define electrical conductivity a [units of amperes per volt-meter 
[A/(V-m)] or mhos/meter], and magnetic conductivity a* [units of volts per 
ampere-meter [V/(A-m)] or ohms/meter]. In this case (12.38) and (12.39) take 
on the more general forms. 

O TJ 

Vx£=-u a*~H, (12.67) 

1 dt 

Vxl = ( h o~E. (12.68) 

dt 

Note that a* is not the complex conjugate of a . In fact, or, a* e R with a > 
for a lossy material (i.e., an imperfect insulator, or a conductor), and a* > 0. As 
before we will assume E possesses only a y component, and H possesses only a z 
component. Since we again have propagation only in the x direction, via (12.44), 
(12.45) in (12.67), and (12.68), we have 

dE y dH z 

— - = -M — --a*H z , (12.69) 

dx dt z 

dH z dEy 

-—S = € —± + crE y . (12.70) 

dx dt 

If E y — E y (x, t) possesses a term that propagates only to the right, then (using 
phasors again) 

E = E y (x, t)y = E e j(Ut e~ yx y (12.71) 

for some suitable propagation constant y. For suitable characteristic impedance Z, 
we must have 

H = H z (x, t)z = -E e jat e- yx z. (12.72) 

We may use (12.69) and (12.70) to determine y and Z. Substituting (12.71) and 
(12.72) into (12.69) and (12.70) and solving for y and Z yields 

z i = J_^ _ y 2 = (_ / - a)e + .)(jf <w/Lt +cr *). (12.73) 

ywe + er 

How to handle the complex square roots needed to obtain Z and y will be dealt 
with below. Observe that the equations in (12.73) reduce to the previous cases 
(12.59), and (12.63) when a — a* — 0. It is noteworthy that when we have the 
condition 

(12.74) 



a a 



then we have 



Z 2 = i— = ^1 %- = -. (12.75) 
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Condition (12.74) is what makes the creation of a PML possible, as will be con- 
sidered later in this section, and will be demonstrated in Section 12.5. 

Now we must investigate what happens to waves when they encounter a sudden 
change in the material properties, specifically, an interface between layers. This 
situation is depicted in Fig. 12.3. Assume that medium 1 has physical parameters 
€\, (JLi, o\, and erj*, while medium 2 has physical parameters 62, M2> ff 2, an d o^- 
The corresponding characteristic impedance and propagation constant for medium 
1 is thus [via (12.73)] 



)(o\i\ +erf 



J \ 



jcoei +a t 
while for medium 2 we have 

2 _ j(Ofl 2 + 02 



Y\ ={j(0€i+ai){jcotn +cr*), 



y 2 = U^ 2 + (T2)U^^2 +^2*)' 



(12.76) 



ja>e 2 + 02 
In Fig. 12.3 for some constants E and H we have for the incident field 



(12.77) 



E t - E t y = Ee 



l^t-Yix 



>'. 



Hi = H t z = H 



jojt-yix-. 



zO 

(Out of the page) 



0"< 

Incident 
wave 



H ' 

Reflected 
wave 



Medium 1 




Interface (boundary) between 
two different media 



///////. 



Medium 2 




Figure 12.3 A plane wave normally incident on an interface (boundary) between two 
different media. The magnetic field components are directed orthogonal to the page. The 
interface is at x = and is the yz— plane. 
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so that 

dH t dE t 

— jcoHi, = —y\Ei, 

dt dx 



and via (12.69) 
implying that 



-y\Ei = -jcofxiHi - a* Hi 

E t _ jo)ji\+ a I 



Zi. (12.78) 



Hi ki 

Similarly, for the transmitted field, we must have 

E t jwu2 + erf 

— = — — ^ = Z 2 . (12.79) 

H t y 2 

But, for the reflected field, again for suitable constants E', and H' we have 

E r = E r y = E' <!***€**$, H r = H r z = H'e' mt e nx z 

so that 

3// r 3£ r 

-— = jwH r , -— = Y\E r , 

at ax 



and via (12.69) 
implying 



]/i£ r = -ja>ixiH r - a*H r 
E r jco/xi + CTj" 



//,- Kl 



= -Zi. (12.80) 



The electric and magnetic field components are tangential to the interface, and so 
must be continuous across it. This implies that at x — 0, and for all t (in Fig. 12.3) 

Hi + H r = H t , E t + E r = E t . (12.81) 

If we substitute (12.78)-(12.80) into (12.81), then after a bit of algebra, we have 

E, 2Z 2 



(12.82) 



and 



It is easy to confirm that 



E t Z 2 + Zy 

E r Z2 — -Zi 
E~~ Z2 + Z1 

x = l + p. (12.84) 



p. (12.83) 
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We call t the transmission coefficient from medium 1 into medium 2, and p is the 
reflection coefficient from medium 1 into medium 2. The coefficients x and p are 
often called Fresnel coefficients, especially in the field of optics. 

If a* — then from (12.73) the propagation constant for a sinusoidal wave in 
a conductor is obtained from 



2 2 

y — —fJ,€a> + ja>fia, 



(12.85) 



where fiew 2 > and cojia > 0. We may express y 2 in polar form: y 2 — r\e^ x . 
But y — re' 6 — [rie j6l ] l/2 . In general, if z — re-' 6 , then 



z l/2 = r l/2 e J9/2^ 



z l/2 = r l/2 e ;(0/2+7r)^ 



(12.86) 



From (12.85) 



n — \y 2 \ — m\i\l a 2 + e 2 a> 2 , 0\ — h tan l (-co) . 

2 \<j I 



(12.87) 



Consequently 



Y 



±W^G 2 + € 2 w 2 ] l l 2 e^ + 2^- i ^l°)\ 



(12.88) 



A special case is the perfect insulator (perfect dielectric) for which a — 0. Since 
tan _1 (oo) = 7r/2, Eq. (12.88) reduces to y = zkj^/JZea) — ±j/S. More generally 
(a not necessarily zero) we have y — ±(a + jfi), where 



a — [ajfiVcr 2 + € 2 co 2 ] ' cos 
P = [co/j,y/a 2 + e 2 co 2 ] l/2 sin 



Tt 1 



tan 



h - tan 

4 2 






(12.89a) 
(12.89b) 



Example 12.3 Assume that /x = /xo, e = €o, and that cr = 0.0001 mhos/meter. 
For co = 2nf [/ is the sinusoid's frequency in Hertz (H z )] from (12.89), we obtain 
the following table: 



/(Hz) 



1 x 10 b 


.015236 


0.025911 


1 x 10 7 


.018762 


0.210423 


1 x 10 8 


.018836 


2.095929 


1 x 10 9 


.018837 


20.958455 



Keeping the parameters the same, except that now a — 0.01 mhos/meter, we 
have the following table: 
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/ (Hz) 


a 


P 


1 x 10 6 


.198140 


0.199245 


1 x 10 7 


.611091 


0.646032 


1 x 10 8 


1.523602 


2.591125 


1 x 10 9 


1.876150 


21.042254 



In general, from (12.89a) we have a > 0. If in Fig. 12.3 medium 1 is free space 
while medium 2 is a conductor with < a < oo, then E t has the form [using 
(12.82)] for x > 

E t = rEe jmt e- y2X y = rEe ja " ' e~ aj - x e~ ihx y . (12.90) 

Of course, we also have E, — Ee^ wt e~^ lX y for x < since y\ — jfii (i-e-, oti = 
in free space), and E r — pEe Jcol e'^ lX y forx < [using (12.83)]. Since a 2 > 0, the 
factor e~ a2X will go to zero as x -> oo. The amplitude of the wave must decay as 
it progresses from free space (medium 1) into the conductive medium (medium 2). 
The rate of decay certainly depends on the size of a 2 . 

Now suppose that medium 1 is again free space, but that medium 2 has both 
<r 2 > and a^ > such that condition (12.74) holds with /x 2 = Mo> an d e 2 — e o> 
specifically 



-±- = — (12.91) 



which implies [via (12.75)] that Z 2 = Vmo/^o- Since medium 1 is free space Z\ — 

VMo/eo, loo. The reflection coefficient from medium 1 into medium 2 is [via 

(12.83)] 

Z 2 — Z\ Zq — Zq 

p — = = 0. 

Z 2 + Z\ 2Zq 

When wave Ej in medium 1 encounters the interface (at x = in Fig. 12.3), there 
will be no reflected component, that is, we will have E r — 0. From (12.73) we 
obtain 

yl = (or 2 or 2 * - w 2 /x eo) + jw(a 2 IMo + or 2 *eo), (12.92) 

and we select the medium 2 parameters so that for y 2 = a 2 + jfii we obtain a 2 > 0, 
and a 2 is large enough so that the wave is rapidly attenuated in that e~ aiX is small 
for relatively small x. In this case we may define medium 2 to be a perfectly 
matched layer (PML). It is perfectly matched in the sense that its characteristic 
impedance is the same as that of medium 1, thus eliminating reflections at the 
interface. Because it is lossy, it absorbs radiation incident on it. The layer dissipates 
energy without reflection. It thus simulates the walls of an anechoic chamber. In 
other words, an anechoic chamber has walls that approximately realize condition 
(12.74). The necessity to simulate the walls of an anechoic chamber will become 
clearer in Section 12.5 when we look at the FDTD method. 
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Finally, we remark on the similarities between the vibrating string problem 
and the problem considered here. The analytical solution method employed in 
Section 12.3.1 was separation of variables, and we have employed the same ap- 
proach here since all of our electromagnetic field solutions are of the form u(x,t) — 
X(x)T(t). The main difference is that in the vibrating string problem we have 
boundary conditions defined by the ends of the string being tied down somewhere, 
while in the electromagnetic wave propagation problem as we have considered it 
here there are no boundaries, or rather, the boundaries are at x — ±00. 



12.4 THE FINITE-DIFFERENCE (FD) METHOD 

We now consider a classical approach to the numerical solution of (12.9) that we 
call the finite -difference (FD) method. Note that the method to follow is by no means 
the only approach. Indeed, the FDTD method to be considered in Section 12.5 is 
an alternative, and there are still others. 
Following (12.5) 

d 2 u(x k , t n ) _ u(x k , t n+ i) - 2u(x k , t n ) + u(x k , t n -\) x 2 3 4 u(x k , rj n ) ..„, 
dt 2 - T 2 12 dt 4 ( } 

for some r\ n e [t n -\, t n +\\, and 

d 2 u(x k , t n ) _ u(x k+ i, t n ) - 2u(x k , t n ) + u(x k -i, t n ) h 2 d 4 u(t; k , t n ) ,,„„., 
dx 2 h 2 12 dx 4 

for some % k € [x k -\, x k+ \], where 

x k — kh, t n — nx (12.95) 

forfc = 0, l,...,M,and« € Z+. On substitution of (12.93) and (12.94) into (12.9), 
we have 

u(x k , t n+ i) - 2u(x k , t n ) + u(x k , t n _x) _ 2 u{x k+ i, t n ) - 2u(x k , t„) + u(x k -i, t n ) 
x 2 ° h 2 

1 



12 



2 d^u(x k ,rj n ) 2 ,2 du ^k,t n ) 

t ; c h -, 

3f 4 dx 4 



(12.96) 



=e kj , 



where e kt „ is the local truncation error. Since u k ,« ^ u(x k , t n ) from (12.96), we 
obtain the difference equation 

Uk,n + \ ~ 2uk,n + Uk,n-l ~ e 2 u k+ \, n + 1e 2 u k , n - e 2 u k -] :n = 0, (12.97) 

where 

e = T T c (12.98) 
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is sometimes called the Courant parameter. It has a crucial role to play in deter- 
mining the stability of the FD method (and of the FDTD method, too). If we solve 
(12.97) for Uk, n +i we obtain 

Uk,n+l = 2(1 - e 2 )ll k , n + e 2 (uk+l,n + Uk-l,n) ~ Uk,n-1, (12.99) 

where k =1,2, , . . , M — 1, andn = 1, 2, 3, Equation (12.99) is the main recur- 
sion in the FD algorithm. However, we need to account for the initial and boundary 
conditions in order to initialize this recursion. Before we consider this matter note 
that, in the language of Section 10.2, the FD algorithm has a truncation error of 
0(x 2 + h 2 ) per step [via e k ,n m (12.96)]. 

Immediately on applying (12.10), since L — Mh, we have 

m ,« = UM,n — for n e Z + . (12.100) 

Since from (12.11) u(x, 0) = f(x), we also have 

«*,<> = /(**) (12.101) 

for k = 0, 1, . . . , M. From (12.101) we have ut,o, but we also need ut,i [consider 
(12.99) for n — 1]. To obtain a suitable expression first observe that from (12.9) 

d 2 u(x,t) T?) 2 u{x,t) d 2 u(x,0) ,3 2 m(x,0) , n -. 

V— = c 2 ~^^ -f- = c 2 K -^-L = c 2 f i2 \x). (12.102) 

dt 2 dx 2 dt 2 dx 2 

Now, on applying the Taylor series expansion, we see that for some \i e [0, t\\ — 
[0, t] (and /i may depend on x) 

du(x, 0) 1 2 3 u(x,0) 1 3 3 m(x, /x) 
u(x,t 1 ) = u(x,0) + T^^ + -T —^- + J x — gp— . 

and on applying (12.11) and (12.102), this becomes 

1 i o n\ 1 1 3 u(x, Li) 

u(x, fi) = u(x, 0) + xg{x) + -r 2 c 2 f i2) (x) + -r 3 -^— . (12.103) 

2 6 dt s 

In particular, for x — x k , this yields 

1 T O CTl 1 1 9 life, Uit) 

«(**, fi) = u(x k , 0) + rgfe) + -t 2 c 2 f (2) (x k ) + -t 3 ' * . (12.104) 

2 6 3f J 

If f(x) e C 4 [0, L], then, for some & e [jc^-i, X£+i], we have 

/(2)fo) = /fa +1 )-2/(* t ) + /(**-l) _ g /( 4) fe) . (12 . 105) 
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Since u(x k ,0) = f{x k ) [recall (12.11) again], and if we substitute (12.105) into 
(12.104), we obtain 



,2 ,.2 



1 c z t" 
u(x k , h) = f(x k ) + xg{x k ) + -—j-[f(x k+ i) - 2f(x k ) + f(xk-i)] 



0(r 3 + r 2 h 2 ). 



(12.106) 



Via (12.98) this yields the required approximation 

«*,i = f(xk) + tg(x k ) + je 2 [f(x k+ i) - 2f(x k ) + f(x k -\)] 



or 



Uk,\ — (1 - e 2 )f(x k ) + je 2 [f(x k+i ) + f(xk-i)] + rg(x k ). 



(12.107) 



Taking account of the boundary conditions (12.100), Eq. (12.99) can be expressed 
in matrix form as 



u\, n +\ 

U2,n + l 

UM-2,n+l 

u M-\,n+\ 



2(1 -e z ) 

„2 






M2,n 

u M-2,n 
UM-l,n 



e 1 

2(1 - e 2 ) e 2 






"2,n-l 



u M-2,n-l 
UM-l,n-l 






2(1 - e 2 ) 



2(1 -e 2 ) 



(12.108) 



This matrix recursion is run for n — 1, 2, 3, . . ., and the initial conditions are pro- 
vided by (12.101) and (12.107). 

We recall from Chapter 10 that numerical methods for the solution of ODE IVPs 
can be unstable. The same problem can arise in the numerical solution of PDEs. 
In particular, as the FD method is effectively an explicit method, it can certainly 
become unstable if h and r are inappropriately selected. 

As noted by others [2, 13], we will have 

lim u k n — u(hk, xri) 

ft,T->0 ' 

provided that < e < 1. This is the famous Courant-Friedrichs-Lewy (CFL) con- 
dition for the stability of the FD method, and is originally due to Courant et al. 
[23]. The special case where e — 1 is interesting and easy to analyze. In this case 
(12.99) reduces to 

Uk,n+l = Uk+l,n ~ "A:,«-l + «/t-l,n- (12.109) 
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We recall that u(x,t) has the form 

u(x, t) — v(x — ct) + w{x + ct) 

[see (12.50)]. Hence u(xk, t n ) — v(x k — ct n ) + w(xk + ct n ). Observe that, since 
h — ex (as e — 1), we have 

u(xt+i, t n ) - u(x k , t n -\) + u(xk-\,t n ) — v(xk + h - ct„) + w{xk +h + ct n ) 

— v(xk — ct n + ct) — w(xk + ct n — ct) 
+ v(xk — h — ct n ) + w(xk — h + ct n ) 
= v(x.k — h — ct n ) + w(xk + h + ct„) 

— v(Xk — Ct n ~ CT ) + w ( x k + ct n + cr ) 

= v(x k - ct n+ \) + w(x k + ct„ + i) 
= u(x k ,t n+ i), 

or 

u(xk, t n +i) = u(x k +\, t n ) - u(Xk, t n -\) + u(x k -i, t n ). (12.110) 

Equation (12.110) has a form that is identical to that of (12.109). In other words, 
the algorithm (12.109) gives the exact solution to (12.9), but only at x — hk and 
t — xn with h — ex (which is a rather restrictive situation). 

A more general approach to error analysis that confirms the CFL condition is 
sometimes called von Neumann stability analysis [2]. We outline the approach as 
follows. We begin by defining the global truncation error 

€k,n = u(Xk,t n ) ~U k ,n- (12.111) 

Via (12.96) 

u(x k ,t n+ i) - 2u(x k , t n ) + u(x k , t n -i) - e 2 [u(xk + \,t n ) -2u(x k ,t n ) + u(x k -\,t n )] 
= r 2 e k ,n- (12-112) 

If we subtract (12.97) from (12.112) and simplify the result using (12.111), we 
obtain 

£k,n+i = 2(1 - e 2 )e k ,n + e 2 [e k +\, n + tk-i,n] ~ e k ,n-i + r 2 e k , n (12.113) 

for k = 0, 1, . . . , M (L — Mh), and n = 1,2,3, .... Equation (12.113) is a two- 
dimensional difference equation for the global error sequence (€k, n )- The term 
x 2 e k ,n is & forcing term, and if u(x,t) is smooth enough, the forcing term will 
be bounded for all k and n. Basically, we can show that the CFL condition < 
e < 1 prevents liirin^oo \€k,n\ — °° f° r a U k — 0, 1, . . . , M. Analogously to our 
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stability analysis approach for ODE IVPs from Chapter 10, we may consider the 
homogeneous problem 

€k,n+\ = 2(1 - e 2 )e kjl + e 2 [e k+hn + e k -i, n ] - €k,n-u (12.114) 

which is just (12.113) with the forcing term made identically zero for all k and 
n. In Section 12.3.1 we learned that separation of variables was a useful means 
to solve (12.9). We therefore believe that a discrete version of this approach is 
helpful at solving (12.114). To this end we postulate a typical solution of (12.114) 
of the form 

e k n — exp[jakh + pnr] (12.115) 

for suitable constants a e R and fi e C. We note that (12.115) has similarities to 
(12.28) and is really a term in a discrete form of Fourier series expansion. We also 
see that 

\€k,n\ = \exp(finx)\ = \s n \. 

Thus, if \s\ < 1, we will not have unbounded growth of the error sequence (e k , n ) 
as n increases. If we now substitute (12.115) into (12.114), we obtain (after sim- 
plification) the characteristic equation 

s 2 - [2(1 -e 2 ) + 2cos(ah)e 2 ]s + 1 = 0. (12.116) 



=2b 



Using the identity 2sin 2 x = 1 — cos(2x), we obtain 



l-2e 2 sin 2 ( — I . (12.117) 



It is easy to confirm that \b\ < 1 for all e such that < e < 1 because < 
sin 2 (ah/2) < 1 for all all ah e R. We note that s 2 - lbs + 1 = for s — S\,S2, 
where 



s\ 



b + y/b 2 - 1, s 2 = b-y/b 2 - 1. (12.118) 



If \b\ > 1, then |^| > 1 for some k e {1, 2}, which can happen if we permit e > 1. 
Naturally we reject this choice as it yields unbounded growth in the size of e k ,n 
as n — > oo. If \b\ < 1, then clearly \s k \ = 1 for all k. (To see this, consider the 
product ^i^2 = Si*i = ki I 2 -) This prevents unbounded growth of e kn . Thus, we 
have validated the CFL condition for the selection of Courant parameter e (i.e., we 
must always choose e to satisfy < e < 1). 

Example 12.4 Figure 12.4 illustrates the application of the recursion (12.108) 
to the vibrating string problem of Example 12.1. The simulation parameters are 
stated in the figure caption. The reader should compare the approximate solution 
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0.05 




-0.05 



Time (m) 







02 Position (kh) 



Figure 12.4 FD method approximate solution to the vibrating string problem. A mesh 



plot of «£ „ as given by Eq. (12.108) for the parameters L = \,c/L 



Additionally, h 
simulation. 



and H 



io- 



0.05 and r =0.1, which meets the CFL criterion for stability of the 



of Fig. 12.4 to the exact solution of Fig. 12.2. The apparent loss of accuracy as the 
number of time steps increases (i.e., with increasing nx) is due to the phenomenon 
of numerical dispersion [22], a topic considered in the next section in the context 
of the FDTD method. Of course, simulation accuracy improves as h, x — > for 
fixed T = Nx, and L = Mh. 



12.5 THE FINITE-DIFFERENCE TIME-DOMAIN (FDTD) METHOD 

The FDTD method is often attributed to Yee [14]. It is a finite-difference scheme 
just as the FD method of Section 12.4 is a finite-difference scheme. However, it is 
of such a nature as to be particularly useful in solving hyperbolic PDEs where the 
boundary conditions are at infinity (i.e., wave propagation problems of the kind 
considered in Section 12.3.2). 

The FDTD method considers approximations to H z (x, t) and E y (x, t) given by 
applying the central difference and forward difference approximations to the first 
derivatives in the PDEs (12.69) and (12.70). We will use the following notation 
for the sampling of continuous functions such as f(x, t): 



fk, n « f(kAx,nAt), f 



k+ 



f{{k+\)Ax,{n 



\)At) 



(12.119) 



(so Ax replaces h, and At replaces r here, where h and r were the grid spacings 
used in previous sections). For convenience, let E — E y and H — H z (i.e., we drop 
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the subscripts on the field components). We approximate the derivatives in (12.69) 
and (12.70) specifically according to 



dH ^ 

~dT " 


1 


(12.120a) 


dE ^ 


1 

s —[Ek,n + l — Ejc^n], 


(12.120b) 


dH ^ 

dx 


1 


(12.120c) 


dE ^ 
dx 


1 
s —[Ek+\,n — Ek,n\- 


(12.120d) 



Define e k — e(kAx), ji k — p((k + \)Ax), a k — a(kAx), and a k * — a*((k + i) 
Ax), which assumes the general situation where the material parameters vary with 
x e [0, L] (computational region). Substituting these discretized material parame- 
ters, and (12.120) into (12.69) and (12.70), we obtain the following algorithm: 



H, 



1 - -*- At 



H, 



k+i,,n- 



1 At 

: — [Ek+l,n — Ek, n ] 

Hk Ax 



Ek,n+1 






At 



&k,n 



1 At 

€k Ax 



[H t 



H, 



(12.121a) 
(12.121b) 



This is sometimes called the leapfrog algorithm. The dependencies between the esti- 
mated field components in (12.121) are illustrated in Fig. 12.5. If we assume that 



H(-\Ax, t) = H((M + \)Ax, f) = 



(12.122) 



Time (f) 




x=0 



M-1 



ooo 



Computational region 




x=L 



Figure 12.5 An illustration of the dependencies between the approximate field components 
given by (12.121a,b); the lines with arrows denote the "flow" of these dependencies [O 
electric field component (E y ); □ magnetic field component (H z )]. 
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for all t e R + , then a more detailed pseudocode description of the FDTD algo- 
rithm is 

H i i := for k = 0, 1 M - 1 ; 

E/( j0 := for /c = 0, 1 , . . . , M; 
for n := to N — 1 do begin 

E m>n := Eo sin(<wnAf); {0 < m < M] 

for /( := to M - 1 do begin 



W 1 1 

end; 



1-^Af 



H /f+ . 2 -,n-±~w^ [E ' f + 1 .''' 



for /( := to M do begin 

& n+ 1 := Pi- ^Afl &,„-!-££ [H i i -H 

end; 
end; 



2 > " "t 2 ^ — 2 ' ^ ~^~ 2 



The statement E mn :— Eosm(amAt) simulates an antenna that broadcasts a 
sinusoidal electromagnetic wave from the location x — mAx e (0, L). Of course, 
the antenna must be located in free space. 

The FDTD algorithm is an explicit difference scheme, and so it may have 

stability problems. However, it can be shown that the algorithm is stable provided 

we have 

Af / Ax \ 

e = c < 1 or Af < , (12.123) 

Ax \ c J 

Thus, the CFL condition of Section 12.4 applies to the FDTD algorithm as well. 

A justification of this claim appears in Taflove [8]. A MATLAB implementation 

of the FDTD algorithm may be found in Appendix 12.A (see routine FDTD.m). In 

this implementation we have introduced the parameters s x and s t (0 < s x , s t < 1) 

such that 

Ax 

Ax = ^A , At = s t • (12.124) 

c 

Clearly, cAt/Ax — s t , and so the CFL condition is met. Also, spatial sampling is 
determined by Ax = s x Xq, which is some fraction of a free-space wavelength Xq 
[recall (12.55)]. Note that the algorithm simulates the field for all t e [0, T], where 
T — NAt. If the wave is propagating only through free space, then the wave will 
travel a distance 

D = cT = Ns,s x X Q , (12.125) 

that is, the distance traveled is Ns,s x free-space wavelengths. Since L = Ms x Xq 
(i.e., the computational region spans Ms x free-space wavelengths), this allows us 
to make a reasonable choice for A^. 

A problem with the FDTD algorithm is that even if the computational region is 
only free-space, a wave launched from location x = mAx € (0, L) will eventually 
strike the boundaries at x = and/or x — L, and so will be reflected back toward 
the source. These reflections will cause very large errors in the estimates of H, and 
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E. But we know from Section 12.3.2 that we may design absorbing layers called 
perfectly matched layers (PMLs) that suppress these reflections. 

Suppose that the PML has physical parameters /i,e,a, and a*, then, from 
(12.73), the PML will have a propagation constant given by 



y — (era* — a> /xe) + ja>(cr /x + cr*e). 



(12.126) 



If we enforce the condition (12.74), namely 



a ' 



a 



(12.127) 



then (12.126) becomes 



y 2 = -((T 2 - OJ 2 € 2 ) + 2JCOGIM. 



(12.128) 



If we enforce a — co e < 0, then y — a + j/3, where 






4 



-Tan 



2^2 



co i e 



2&)CT6 



(12.129) 



Equation (12.129) is obtained by the same arguments that yielded (12.89). As 
a > 0, then a wave on entering the PML will be attenuated by a factor e~ ax , 
where x is the depth of penetration of the wave into the PML. A particularly 
simple choice for a is to let a 2 = co 2 e 2 , in which case 



Since co — l^-c and c — ' 



(12.130) as 



/M0<?0 



a^co^JU. (12.130) 

with /i = /Xrfio and e — e r €o, we can rewrite 



2n 

a = siller — - 
>-0 



(12.131) 



If we are matching the PML to free space, then ji r — e r — 1, and so a = 
which case 

„—OLX 



In 



in 



e A o 



(12.132) 

If the PML is of thickness x = 2A , then, from (12.132) we have e~ ax = e~ An % 
3.5 x 10~ 6 . A PML that is two free-space wavelengths thick will therefore absorb 
very nearly all of the radiation incident on it at the wavelength Ao. Since we have 
chosen a 2 — co 2 e 2 , it is easy to confirm that 



In e r t 2it 

a = , a = — l^rZo 

Xq Zq Xq 



(12.133) 
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4 5 6 7 

Distance (xA ) 



10 



Figure 12.6 Typical output from FDTD.m (see Appendix 12.A) for the system described in 
Example 12.5. The antenna broadcasts a sinusoid of wavelength Xq = 500 nm (nanometers) 
from the location x = 3A.Q. The transmitted field strength is Eq = 1 V/m. 



[via (12.127)], where we recall that Zq = VTW^o- Routine FDTD.m in Appen- 
dix 12. A implements PMLs according to this approach. 



Example 12.5 Figure 12.6 illustrates a typical output from FDTD.m (Appen- 
dix 12.A). The system shown in the figure occupies a computational region of 
length 10Ao (i.e., x e [0, lOArj])- Somewhat arbitrarily we have Xq — 500 nm (nano- 
meters). The antenna (which, given the wavelength, could be a laser) is located 
at index m — 150 (i.e., is at x — mAx — ms x Xo — 3Xq, since s x — .02). The free- 
space region is for x e (2A,n, 6Ao). The lossless dielectric occupies x e [6Ao, 8Xo], 
and has a relative permittivity of e r — 4. The entire computational region is non- 
magnetic, and so we have /x = /j,q everywhere. Clearly, PML 1 is matched to free 
space, while PML 2 is matched to the dielectric. 

Since e r — 4, according to (12.82), the transmission coefficient from free space 
into the dielectric is 



T = 



2/^r 

V e rf0 


2 


2 


/ M-0 , /MO 

V f,-eo V e o 


l + V^ 7 


3 
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Since Eq = 1 V/m, the amplitude of the electric field within the dielectric must be 

2 



xEq = I V/m. From Fig. 12.6 the reader can see that the electric field within the 



dielectric does indeed have an amplitude of about | V/m to a good approximation. 
From (12.57) the wavelength within the dielectric material is 

1 1 

X — ——Xq — -X.Q. 

Again from Fig. 12.6 we see that the wavelength of the transmitted field is indeed 
close to jXq within the dielectric. 

We observe that the PMLs in Example 12.5 do not perfectly suppress reflections 
at their boundaries. For example, the wave crest closest to the interface between 
PML 2 and the dielectric, and that lies within the dielectric, is somewhat higher 
than it should be. It is the discretization of a continuous space that has lead to these 
residual reflections. 

The theory of PMLs presented here does not easily extend from electromagnetic 
wave propagation problems in one spatial dimension into propagation problems in 
two or three spatial dimensions. It appears that the first truly successful extension of 
PML theory to higher spatial dimensions is due to Berenger [15,16]. Wu and Fang 
[17] claim to have improved the theory still further by improving the suppression 
of the residual reflections noted above. 

The problem of numerical dispersion was mentioned in Example 12.4 in the 
application of the FD method to the simulation of a vibrating string. We conclude 
this chapter with an account of the problem based mainly on the work of Trefethen 
[22]. We will assume lossless propagation, so a — a* = in (12.121). We will also 
assume that the computational region is free space, so p — po, and e — €q every- 
where. If we now substitute E(x, t ) — Eq sin(ct>t — fix) and H(x,t) — Hq sin(&;f — 
fix) into either of (12.121a) or (12.121b), apply the appropriate trigonometric iden- 
tities, and then cancel out common factors, we obtain the identity 

/w A A //SAx\ 
sin =esin . (12.134) 



We may use (12.134) and (12.123) to obtain 

ft) c 2 , 

v„ — — — sin 

p efiAx 



esm 



J Ax 



(12.135) 



which is the phase speed of the wave of wavelength Xo (recall /3 = 2jt/Xo) in the 
FDTD method. For the continuous wave E(x, t) [or, for that matter, H(x, t)], recall 
from Section 12.3.2 that <o/fi — c, so without spatial or temporal discretization 
effects, a sinusoid will propagate through free space at the speed c regardless of its 
wavelength (or, equivalently, its frequency). However, (12.135) suggests that the 
speed of an FDTD-simulated sinusoidal wave will vary with the wavelength. As 
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explained in Ref. 22 (or see Ref. 12), the group speed 



dw d 
"dp 7 ~ lp 



cos 



iPVn) 



(i£) 



1 — e 2 sin 2 



(*£) 



(12.136) 



is more relevant to assessing how propagation speed varies with the wavelength. 
Again, in free space the continuous wave propagates at the group speed % = 

jg(fic) — c. A plot of Vg/c versus An/Ax [with v g /c given by (12.136)] appears 
in Fig. 2 of Represa et al. [19]. It shows that short-wavelength sinusoids travel at 
slower speeds than do long-wavelength sinusoids when simulated using the FDTD 
method. 

We have seen that nonsinusoidal waveforms (e.g., the triangle wave of Exam- 
ple 12.1) are a superposition of sinusoids of varying frequency. Thus, if we use 
the FDTD method, the FD method, or indeed any numerical method to simulate 
wave propagation, we will see the effects of numerical dispersion. In other words, 
the various frequency components in the wave will travel at different speeds, and 
so the original shape of the wave will become lost as the simulation progresses 
in time (i.e., as «Af increases). Figure 12.7 illustrates this for the case of two 
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Figure 12.7 Numerical dispersion in the FDTD method as illustrated by the propagation 
of two Gaussian pulses. The medium is free space. 
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Gaussian pulses traveling in opposite directions. The two pulses originally appeared 
at x = 4A.o, and the medium is free space. For N = 1500 time steps, there is a very 
noticeable error due to the "breakup" of the pulses as their constituent frequency 
components separate out as a result of the numerical dispersion. 

In closing, note that more examples of numerical dispersion may be found in 
Luebbers et al. [18]. Shin and Nevels [21] explain how to work with Gaussian 
test pulses to reduce numerical dispersion. We mention that Represa et al. [19] 
use absorbing boundary conditions based on the theory in Mur [20], which is a 
different method from the PML approach we have used in this book. 



APPENDIX 12.A MATLAB CODE FOR EXAMPLE 12.5 



% permittivity .m 

% This routine specifies the permittivity profile of the 
% computational region [0,L], and is needed by FDTD.m 

% 

function epsilon = permittivity (k,sx,lambdaO,M) 

epsilonO = 8.854185*1e-12; % free-space permittivity 
er1 =4; % relative permittivity of the dielectric 

Dx = sx*lambdaO; % this is Delta x 

x = k*Dx; % position at which we determine epsilon 

L = M*Dx; % location of right end of computational 

% region 

if ((x >= 0) & (x < (L-4*lambda0))) 

epsilon = epsilonO; 
else 

epsilon = er1*epsilon0; 

end; 

q, 
'o 

% permeability .m 

% 

% This routine specifies the permeability profile of the 

% computational region [0,L], and is needed by FDTD.m 



function mu = permeability(k,sx,lambdaO,M) 

muO = 400*pi*1e-9; % free-space permeability 
Dx = sx*lambdaO; % this is Delta x 

x = k*Dx; % position at which we determine mu 

L = M*Dx; % location of right end of computational 

% region 

mu = muO; 

q, 

% econductivity .m 

q, 

% This routine specifies the electrical conductivity profile 
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% of the computational region [0,L], and is needed by FDTD.m 

function sigma = econductivity(k,sx,lambdaO,M) 



epsilonO = 8.854185*1e-12; 

muO = 400*pi*1e-9; 

er1 = 4; 

epsilonl = er1*epsilon0; 

ZO = sqrt(muO/epsilonO) ; 

Dx = sx*lambdaO; 

x = k*Dx; 

L = M*Dx; 



free-space permittivity 

free-space permeability 

dielectric relative permittivity 

dielectric permittivity 

free-space impedance 

this is Delta x 

position at which we determine sigma 

location of right end of computational 

region 



starl = (2*pi/lambda0)*(1/Z0; 

star2 = er1 *star1 ; 

if ((x > 2*lambda0) & (x < L 

sigma = 0; 
elseif (x <= 2*lambda0) 

sigma = starl ; 
elseif (x >= (L - 2*lambda0); 

sigma = star2; 

end; 



; % conductivity of PML 1 (at x = end) 
% conductivity of PML 2 (at x = L end) 
2*lambda0)) 



mconductivity .m 

This routine specifies the magnetic conductivity profile of the 
computational region [0,L], and is needed by FDTD.m 



function sigmastar = mconductivity(k,sx,lambdaO,M) 



epsilonO = 8.854185*1e-12; 

muO = 400*pi*1e-9; 

ZO = sqrt(muO/epsilonO) ; 

Dx = sx*lambdaO; 

x = (k+.5)*Dx; 

L = M*Dx; 



free-space permittivity 

free-space permeability 

free-space impedance 

this is Delta x 

position at which we determine sigmastar 

location of right end of computational 

region 



star = (2*pi/lambda0)* 


'ZO; 








if ((x > 2*lambda0) & 


(x < 


L 


■2' 


*lambda0 


sigmastar 


= 0; 










else 












sigmastar 


= star; 










end; 












% 










FDTD. 



This routine produces the plot in Fig. 12.6 which is associated with 

Example 12.5. Thus, it illustrates the FDTD method. 

The routine returns the total electric field component Ey to the 
caller. 
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function Ey = FDTD 

muO = 400*pi*1e-9; 
epsilonO = 8.854185*1e-12; 
c = 1 /sqrt (muO*epsilonO) ; 

lambdaO = 500; 

lambdaO = lambda0*1e-9; 

betaO = (2*pi) /lambdaO; 

sx = .02; 

st = .10; 

Dx = sx*lambda0; 
Dt = (st/c)*Dx; 



EO = 1 ; 

m = 150; 

omega = betaO*c; 

M = 500; 
N = 4000; 

E = zeros(1 ,M+1 ) ; 
H = zeros(1 ,M+2) ; 



% free-space permeability 
% free-space permittivity 
% speed of light in free-space 

% free-space wavelength of the source 

% in nanometers 

% free-space wavelength of the source 

% in meters 

% free-space beta (wavenumber) 

% fraction of a free-space wavelength 

% used to determine Delta x 

% scale factor used to determine time-step 

% size Delta t 

% Delta x 

% application of the CFL condition 

% to determine time-step Delta t 

% amplitude of the electric field (V/m) 

% generated by the source 

% source (antenna) location index 

% source frequency (radians/second) 

% number of spatial grid points is (M+1 ) 

% the number of time steps in the simulation 

% initial electric field 
% initial magnetic field 



% Specify the material properties in the computational region 
% (which is x in [0,M*Dx]) 

for k = 0:M 

epsilon(k+1) = permittivity(k, sx, lambdaO, M) ; 

ce(k+1) = 1 - Dt*econductivity(k, sx, lambdaO, M) /epsilon(k+1 ) ; 

h(k+1) = 1/epsilon(k+1); 

end; 
h = h*Dt/Dx; 
for k = 0: (M-1) 

mu(k+1) = permeability(k, sx, lambdaO, M) ; 

ch(k+1) = 1 - Dt*mconductivity(k, sx, lambdaO, M) /mu(k+1 ) ; 

e(k+1) = 1/mu(k+1); 

end; 
e = e*Dt/Dx; 

% Run the simulation for N time steps 

for n = 1 :N 

E(m+1) = E0*sin(omega*(n-1 )*Dt) ; % Antenna is at index m 

H(2:M+1) = ch.*H(2:M+1) - e.*(E(2:M+1) - E(1:M)); 

E(1:M+1) = ce.*E(1:M+1) - h.*(H(2:M+2) - H(1:M+1)); 

E(m+1) = EO*sin(omega*n*Dt) ; 

Ey(n,:) = E; % Save the total electric field at time step n 

end; 
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PROBLEMS 

12.1. Classify the following PDEs into elliptic, parabolic, and hyperbolic types 
(or a combination of types). 



(a) 


?>u xx 


+ 5U X y + Uyy 


— x + y 




(b) 


u xx 


~ "XV l" ^Hyy — 


- u x + u 




(c) 


yuxx 


+ Uyy — 






(d) 


y 2 u xx - 2xyu xy + 


2 

A, Lt -y v 





(e) 


4x 2 u 


XX ~T Uyy U 






;.2. Derive (12.5) (both equations). 





12.3. In (12.6) let h — t, and let N — M — 4, and for convenience let f^ n — 
p(xk, yn)/z- Define the vectors 

v = [v u vi,2 Vi,3 v 2A v 2 , 2 v 2 ,3 v 3 ,i v X2 y 3 , 3 ], 

/ = [/l,l /l,2 .A, 3 /2,1 h,2 /2,3 /3,1 /3,2 /3,3]- 

Find the matrix A such that AV — h 2 f. Assume that 
V ,„ = Vi.o = V kA = V 4 ,„ = 
for all £, and n. 

12.4. In (12.6) let h = r = \, and N — M — 4, and assume that p(xk, y n ) — 
for all k, and n. Let xo = yo — 0. Suppose that 

V(0, y) = V(x, 0) = 0, V(x, l) = x, V(l, y) = y. 

Find the linear system of equations for Vk, n with 1 < k, n < 3, and put it 
in matrix form. 

12.5. Recall the previous problem. 

(a) Write a MATLAB routine to implement the Gauss-Seidel method (recall 
Section 4.7). Use your routine to solve the linear system of equations 
in the previous problem. 

(b) Find the exact solution to the PDE 

v xx + y yy = o 

for the boundary conditions stated in the previous problem. 
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(c) Use the solution in (b) to find V(k/4, n/4) at 1 < k, n < 3, and compare 
to the results obtained from part (a). They should be the same. Explain 
why. [Hint: Consider the error terms in (12.5).] 

12.6. The previous two problems suggest that the linear systems that arise in the 
numerical solution of elliptic PDEs are sparse, and so it is worth considering 
their solution using the iterative methods from Section 4.7. Recall that the 
iterative methods of Section 4.7 have the general form 

X (*+D = Bx (k) + f 
[from (4.155)]. Show that 



c<* +1 >-j<*>|| c 

c (*) -*<*-!) || { 



\B\ 



[Comment: Recalling (4.36c), it can be shown that p(A) < \\A\\ p [which is 
really another way of expressing (4.158)]. For example, this result can be 
used to estimate the spectral radius of Bj [Eq. (4.171)].] 

12.7. Consider Eq. (12.9), with the initial and boundary conditions in (12.10) and 
(12.11), respectively. 

(a) Consider the change of variables 

§ = x + ct , r\ — x — ct 
and 0(£, n) replaces u{x,t) according to 

0(1, ij) = u Q(§ + n ), 1(| - 1,)^ . (12.P.1) 

Verify the derivative operator equivalences 

9_3 913_9 9 
dx 9^ dr/ c dt 9f dn 

[Hint' ^L du_dx_ _i_ dudt_ 90 du_ dx_ , 'du_ dt_ -i 

\_nini. a ^ — 3x g ^ -i- 3l 9 |, 9r/ — 3x dr) -I- 3t Sn -i 

(b) Show that (12.9) is replaceable with 

9 2 c 



(12.P.2) 



0. (12.P.3) 



d^dn 
(c) Show that the solution to (12.P.3) is of the form 

and hence 

u(x, t) — P(x + ct) + Q(x - ct), 

where P and Q are arbitrary twice continuously differentiable functions. 
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(d) Show that 



P(x) + Q(x) = /(*), 



>(i), 



)(i)/ 



1 



P w (x)-Q w (x) = -g(x). 
c 



(e) Use the facts from (d) to show that 



1 



"X+Ct 



u(x,t) = -[f(x + ct) + f(x-ct)] + — / g(s)ds. 

2 2c J x _ ct 



12.8. In (12.11) suppose that 



fix) 



H ( L 

— I x d 

d \ 2 



H ( L 

— x \- d 

d \ 2 



0, 



L L 

— < x < \-d 

2 ~ ~ 2 

L L 

d < x < — 

2 ~ ~ 2 

elsewhere 



where < d < L/2. Assume that g(x) — for all x. 

(a) Sketch f(x). 

(b) Write a MATLAB routine to implement the FD algorithm (12.108) for 
computing «^„. Write the routine in such a way that it is easy to change 
the parameters c, H, h, r, N, M, and d. The routine must produce a mesh 
plot similar to Fig. 12.4 and a plot similar to Fig. 12.7 on the same page 
(i.e., make use of the subplot). The latter plot is to be of uu,n versus k. Try 
out your routine using the parameters c = i, h = 0.05, r — 0.025, H = 
0.1, d = L/10, M = 200, and N = 1100 (recalling that L = Mh). Do 
you find numerical dispersion effects? 



12.9. Repeat Example 12.1 for f(x) and g(x) in the previous problem. 

12.10. Example 12.1 is about the "plucked string." Repeat Example 12.1 assuming 
that f{x) — and 



,'(*) 



2V 



2V 



x), 



L 
0<x < - 

~ ~ 2 

L 

— < x < L 

2 ~ ~ 



This describes a "struck string." 



12.11. The MATLAB routines in Appendix 12.A implement the FDTD method, 
and generate information for the plot in Fig. 12.6. However, the reflected 
field component for 21o < x < 6Xq is not computed or displayed. 

Modify the code(s) in Appendix 12. A to compute the reflected field com- 
ponent in the free- space region and to plot it. Verify that at the interface 
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between the free-space region and the dielectric that |p| = A (magnitude of 
the reflection coefficient). Of course, you will need to read the amplitude 
of the reflected component from your plot to do this. 

12.12. Derive Eq. (12.134). 

12.13. Modify the MATLAB code(s) in Appendix 12. A to generate a plot similar 

. 21 



to Fig. 12.7. [Hint: E^q — Eq exp 



( k—m \ 
iw ) 



for k = 0, 1, ..., M is the 



initial electric field. Set the initial magnetic field to zero for all k.] 

12.14. Plot Vg/c versus Xo/Ax for v s given by (12.136). Choose e = 0.1, 0.5, and 
0.8, and plot all curves on the same graph. Use these curves to explain why 
the errors due to numerical dispersion (see Fig. 12.7) are worse on the side 
of the pulse opposite to its direction of travel. 
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13.1 INTRODUCTION 

MATLAB is short for "matrix laboratory," and is an extremely powerful software 
tool 1 for the development and testing of algorithms over a wide range of fields 
including, but not limited to, control systems, signal processing, optimization, image 
processing, wavelet methods, probability and statistics, and symbolic computing. 
These various applications are generally divided up into toolboxes that typically 
must be licensed separately from the core package. 

Many books have already been written that cover MATLAB in varying degrees 
of detail. Some, such as Nakamura [3], Quarteroni et al. [4], and Recktenwald 
[5], emphasize MATLAB with respect to numerical analysis and methods, but 
are otherwise fairly general. Other books emphasize MATLAB with respect to 
particular areas such as matrix analysis and methods (e.g., see Golub and Van 
Loan [1] or Hill [2]). Some books implicitly assume that the reader already knows 
MATLAB [1,4]. Others assume little or no previous knowledge on the part of the 
reader [2,3,5]. 

This chapter is certainly not a comprehensive treatment of the MATLAB tool, 
and is nothing more than a quick introduction to it. Thus, the reader will have to 
obtain other books on the subject, or consult the appropriate manuals for further 
information. MATLAB 's online help facility is quite useful, too. 



13.2 STARTUP 

Once properly installed, MATLAB is often invoked (e.g., on a UNIX workstation 
with a cmdtool window open) by typing matlab, and hitting return. A window 
under which MATLAB runs will appear. The MATLAB prompt also appears: 



MATLAB commands may then be entered and executed interactively. 

If you wish to work with commands in M-files (discussed further below), then 
having two cmdtool windows open to the same working directory is usually desir- 
able. One window would be used to run MATLAB, and the other would be used 

MATLAB is written in C, but is effectively a language on its own. 



An Introduction to Numerical Analysis for Electrical and Computer Engineers, by C.J. Zarowski 
ISBN 0-471-46737-5 © 2004 John Wiley & Sons, Inc. 
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to edit the M-files as needed (to either develop the algorithm in the file or to 
debug it). 



13.3 SOME BASIC OPERATORS, OPERATIONS, AND FUNCTIONS 

The MATLAB command 

>> diary filename 

will create the file filename, and every command you type in and run, and every 
result of this, will be stored in filename. This is useful for making a permanent 
record of a MATLAB session which can help in documentation and sometimes in 
debugging. When writing MATLAB M-files, always make sure to document your 
programs. The examples in Section 13.6 illustrate this. 

MATLAB tends to work in the more or less intuitive way where matrix and/or 
vector operations are concerned. Of course, it is in the nature of this software tool 
to assume the user is already familiar with matrix analysis and methods before 
attempting to use it. 

When a MATLAB command creates a vector as the output from some operation, 
it may be in the form of a column or a row vector, depending on the command. A 
typical MATLAB row vector is 

»x =[111]; 
>> 

The semicolon at the end of a line prevents the printout of the result of the command 
at that line (this is useful in preventing display clutter). If you wish to turn it into 
a column vector, then type: 

>>x = x. ' 
x = 

1 
1 
1 

Making this conversion is sometimes necessary as the inputs to some routines need 
the vectors in either row or column format, and some routines do not care. Routines 
that do not care whether the input vector is row or column make the conversion to 
a consistent form internally. For example, the following command sequence (which 
can be stored in a file called an M-file) converts vector x into a row vector if it is 
not one already: 

» [N,M] = size(x) ; 

» if N -= 1 

» x = x. ' ; 

» end; 

>> 
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In this routine N is the number of rows and M is the number of columns in x. (The 
size command also accepts matrices.) The related command length(x) will return 
the length of vector x. This can be very useful in for loops (below). 
The addition of vectors works in the obvious way: 

» x = [ 1 1 1 ] ; 
» y = [ 2 -1 2 ]; 

» x + y 

ans = 



In this routine, the answer might be saved in vector z by typing >> z — x + y\. 
Clearly, to add vectors without error means that they must be of the same size. 
MATLAB will generate an error message if matrix and vector objects are not 
dimensioned properly when operations are performed on them. The mismatching 
of array sizes is a very common error in MATLAB programming. 
Matrices can be entered as follows: 

» A = [ 1 1 ; 1 2 ] 

A = 

1 1 

1 2 



Again, addition or subtraction would occur in the expected manner. We can invert 
a matrix as follows: 

» inv(A) 

ans = 

2 -1 

-1 1 



Operation det(A) will give the determinant of A, and [L, U] = lu(A) will give the 
LU factorization of A (if it exists). Of course, there are many other routines for 
common matrix operations, and decompositions (QR decomposition, singular value 
decomposition, eigendecompositions, etc.). Compute y — Ax + b: 

» x = [ 1 -1 ].'; 

» b = [ 2 3 ] . ' ; 



>> 



y = A*x + b 
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The colon operator can extract parts of matrices and vectors. For example, to 
place the elements in rows j to k of column n of matrix B into vector x, use 
> > x — B(j : k, n);. To extract the element from row k and column n of matrix A, 
use >> x — A(k, «);. To raise something (including matrices) to a specific power, 
use >> C — A A p, for which p is the desired power. (This computes C — A p '.) 
{Note: MATLAB indexes vectors and matrices beginning with 1.) 
Unless the user overrides the defaults, variables i and j denote the square root 
of-1: 

» sqrt(-1 ) 

ans = 

+ 1 .OOOOi 



Here, i andy are built-in constants. So, to enter a complex number, say, z — 3 — 2j, 
type 

» z = 3 - 2*i; 
Observe 

» x = [ 1 1+i ] ; 
» x' 

ans = 

1 .0000 

1 .0000 - 1 .0000i 



So the transposition operator without the period gives the complex-conjugate 
transpose (Hermitian transpose). Note that besides i and j, another useful built-in 
constant is pi (= it). 

Floating-point numbers are entered as, for instance, \.5e— 3 (which is 1.5 x 
10 -3 ). MATLAB agrees with IEEE floating-point conventions, and so 0/0 will 
result in NaN ("not a number") to more clearly indicate an undefined operation. 
An operation like 1/0 will result in Inf as an output. 



TLFeBOOK 



SOME BASIC OPERATORS, OPERATIONS, AND FUNCTIONS 569 

We may summarize a few important operators, functions, and other terms: 



Relational Operators 


< 


less than 


<= 


less than or equal to 


> 


greater than 


> = 


greater than or equal to 


== 


equal to 


~= 


not equal to 




Trigonometric Functions 



sin sine 

cos cosine 

tan tangent 

asin arcsine 

acos arccosine 

atan arctangent 

atan2 four quadrant arctangent 

sinh hyperbolic sine 

cosh hyperbolic cosine 

tanh hyperbolic tangent 

asinh hyperbolic arcsine 

acosh hyperbolic arccosine 

atanh hyperbolic arctangent 



Elementary Mathematical Functions 



abs absolute value 

angle phase angle (argument of a complex number) 

sqrt square root 
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Elementary Mathematical Functions 


real 


real part of a complex number 


imag 


imaginary part of a complex number 


conj 


complex conjugate 


rem 


remainder or modulus 


exp 


exponential to base e 


log 


natural logarithm 


log 10 


base- 10 logarithm 


round 


round to nearest integer 


fix 


round toward zero 


floor 


round toward — oo 


ceil 


round toward +oo 



In setting up the time axis for plotting things (discussed below), a useful com- 
mand is illustrated by 

» y = [0: .2:1] 
y = 

0.2000 0.4000 0.6000 0.8000 1.0000 



Thus, [x :y :z ] creates a row vector whose first element is x and whose last element 
is z (depending on step size y), where the elements in between are of the form x 
+ ky (where k is a positive integer). 

It can also be useful to create vectors of zeros: 

>> zeros(size( [1:4])) 







Or, alternatively, a simpler way is 

>> zeros(1 ,4) 
ans = 
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Using "zeros(n,m)" will result in an n x m matrix of zeros. Similarly, a vector 
(or matrix) containing only ones would be obtained using the MATLAB function 
called "ones." 



13.4 WORKING WITH POLYNOMIALS 

We have seen on many occasions that polynomials are vital to numerical analysis 
and methods. MATLAB has nice tools for dealing with these objects. 

Suppose that we have polynomials P\(s) — s + 2 and P2(s) = 3.v + 4 and wish 
to multiply them. In this case type the following command sequence: 

» P1 = [ 1 2 ] ; 

» P2 = [ 3 4 ] ; 
» conv(P1 ,P2) 

ans = 

3 10 8 



This is the correct answer since Pi(s)P2(s) — 3s 2 + 10s + 8. From this we see 
polynomials are represented as vectors of the polynomial coefficients, where the 
highest-degree coefficient is the first element in the vector. This rule is followed 
pretty consistently. (Note: "conv" is the MATLAB convolution function, so if you 
don't already know this, convolution is mathematically essentially the same as 
polynomial multiplication.) 

Suppose P(s) = s 2 + s — 2, and we want the roots. In this case you may type 

» P = [ 1 1 -2 ] ; 
» roots(P) 



-2 

1 



MATLAB (version 5 and later) has "mroots," which is a root finder that does a 
better job of computing multiple roots. 

The MATLAB function "polyval" is used to evaluate polynomials. For example, 
suppose P(s) — s 2 + 3s + 5, and we wanted to compute P(— 3). The command 
sequence is 

» P = [ 1 3 5 ] ; 
» polyval(P, -3) 
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ans = 

5 
>> 

13.5 LOOPS 

We may illustrate the simplest loop construct in MATLAB as follows: 

» t = [ : . 1 : 1 ] ; 

>> for k = 1 :length(t) 

» x(k) = 5*sin( (pi/3) * t(k) ) + 2; 

» end; 

>> 

This command sequence computes x(t) — 5 sin(jt) + 2 for t — O.Ik, where k — 
0, 1, . . . , 10. The result is saved in the (row) vector x. However, an alternative 
approach is to vectorize the calculation according to 

» t = [0: .1:1]; 

» x = 5*sin( pi*t/3 ) + 2*ones(1 , length(t) ) ; 



This yields the same result. Vectorizing calculations leads to faster code (in terms 
of runtime). 

A potentially useful method to add (append) elements to a vector is 

» x = []; 

» for k = 1:2:6 

>> x = [ x k ] ; 

» end; 

» x 



where x — [ ] defines x to be initially empty, while the for loop appends 1,3, and 
5 to the vector one element at a time. 

The format of numerical outputs can be controlled using MATLAB fprintf 
(which has many similarities to the ANSI C fprintf function). For example 

» for k = 0:9 

fprintf ( '%12.8f \n' ,sqrt(k)) ; 
end; 
0.00000000 
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1 .00000000 
1 .41421356 
1.73205081 
2.00000000 
2.23606798 
2.44948974 
2.64575131 
2.82842712 
3.00000000 



The use of a file identifier can force the result to be printed to a specific file instead 
of to the terminal (which is the result in this example). MATLAB also has save 
and load commands that can save variables and arrays to memory, and read them 
back, respectively. 

Certainly, for loops may be nested in the expected manner. Of course, MATLAB 
also supports a "while" statement. For information on conditional statements (i.e., 
"if" statements), use ">> help if." 



13.6 PLOTTING AND M-FILES 

Let's illustrate plotting and the use of M-files with an example. Note that M-files 
are also called script files (use script as the keyword when using help for more 
information on this feature). 

As an exercise the reader may wish to create a file called "stepH.m" (open and 
edit it in the manner you are accustomed to). In this file place the following lines: 



% stepH.m 

(X. 

% This routine computes the unit-step response of the 
% LTI system with system function H(s) given by 



% H(s) = 

% s'2 + 3s + K 

(X. 

% for user input parameter K. The result is plotted. 



function stepH(K) 

b = [ K ] ; 
a = [ 1 3 K ]; 

elf % Clear any existing plots from the screen 

step(b,a); % Compute the step response and plot it 
grid % plot the grid 
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This M-file becomes a MATLAB command, and for K — 0.1 may be executed 
using 

» stepH( .1) ; 
>> 

This will result in another window opening where the plot of the step response will 
appear. To save this file for printing, use 

>> print -dps filename.ps 

which will save the plot as a postscript file called "filename.ps." Other printing for- 
mats are available. As usual, the details are available from online help. Figure 13.1 
is the plot produced by stepH.m for the specified value of K. 

Another example of an M-file that computes the frequency response (both mag- 
nitude and phase) of the linear time-invariant (LTI) system with Laplace transfer 
function is 



H(s) 



1 



Q. 

E 
< 



1 
0.9 
0.8 
0.7 
0.6 
0.5 
0.4 
0.3 
0.2 
0.1 



I I I \- I 



30 60 90 120 

Time (seconds) 



150 



180 



Figure 13.1 Step response: typical output from stepH.m. 
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If you are not familiar with Laplace transforms, recall phasor analysis from basic 
electric circuit theory. You may, for instance, interpret H(ja>) as the ratio of two 
phasor voltages (frequency a>), such as 

„,. . V2<Jo>) 
Vi(jco) 

The numerator phasor V^O&O is the output of the system, and the denominator 
phasor Vi(ja>) is the input to the system (a sinusoidal voltage source). 



% freqresp.m 

% 

% This routine plots the frequency response of the Laplace 

% transfer function 



% H(s) = 

% s"2 + .5s + 1 

% 

% The magnitude response (in dB) and the phase response (in 
% degrees) are returned in the vectors mag, and pha, 
% respectively. The places on the frequency axis where the 
% response is computed are returned in vector f . 



function [mag, pha, f] = freqresp 

b = [ 1]; 

a = [ 1 .5 1 ] ; 

w = logspace( -2, 1 ,50) ; % Compute the frequency response for i0"(-2) to 10"1 

% radians per second at 50 points in this range 
h = f reqs(b,a,w) ; % h is the frequency response 

mag = abs(h); % magnitude response 
pha = angle(h); % phase response 

f = w/(2*pi); % setup frequency axis in Hz 

pha = pha*180/pi; % phase now in degrees 

mag = 20*log10(mag) ; % magnitude response now in dB 

elf 

subplot(211 ) , semilogx(f ,mag, ' - ' ) , grid 
xlabel(' Frequency (Hz) ') 
ylabel(' Amplitude (dB) ') 
title (' Magnitude Response ') 

subplot(212) , semilogx(f ,pha, ' - ' ) , grid 
xlabel(' Frequency (Hz) ') 
ylabel(' Phase Angle (Degrees) ') 
title(' Phase Response ') 

Executing the command 

» [mag,phase,f ] = freqresp; 
» 
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Figure 13.2 The output from freqresp.m: (a) magnitude response; (b) phase response. 



will result in the plot of Fig. 13.2, and will also give the vectors mag, phase, 
and/. Vector mag contains the magnitude response (in decibels), and vector phase 
contains the phase response (in degrees) at the sample values in the vector/, which 
defines the frequency axis for the plots (in hertz). In other words, "freqresp.m" is a 
Bode plotting routine. Note that the MATLAB command "bode" does Bode plots 
as well. 

Additional labeling may be applied to plots using the MATLAB command text 
(or via a mouse using "gtext"; see online help). As well, the legend statement is 
useful in producing labels for a plot with different curves on the same graph. For 
example 

function ShowLegend 

ul = 2.5*pi; 
11 = -pi/4; 
N = 200; 

dt = (ul - 11)/N; 
for k = 0:N-1 

t(k+1) = 11 + dt*k; 

x(k+1) = exp(-t(k+1)); 

y(k+1) = sin(t(k+1)); 

end; 
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Figure 13.3 Illustration of the MATLAB legend statement. 



subplot(21 1 ) , plot(t,x, ' - ' ,t,y, ' -- ' ) , grid 

xlabelC t ') 

legend)' e"{-t} ', ' sin(t) ') 

When this code is run, it gives Fig. 13.3. Note that the label syntax is similar to 
that in LaTeX [6]. 

Other sample MATLAB codes have appeared as appendixes in earlier chapters, 
and the reader may wish to view these as additional examples. 
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Absolute convergence, 95 
Additive splitting, 179 
Adjoint matrix, 489 
Alternating set, 239 
Ampere's law, 535 
Anechoic chamber, 544 
Appolonius' identity, 35 
Argmin, 39 
ASCII, 339 

Asymptotic expansion, 97 
Asymptotic series, 97-103 
Asymptotic time complexity, 



155 



Two's complement, 57 

Binomial theorem, 85 

Bipolar junction transistor (BJT) 
Base current, 420 
Base-emitter voltage, 420 
Collector current, 420 
Collector-emitter voltage, 420 
Forward current gain, 420 
On resistance, 420 

Bisection method, 292-296 

Block upper triangular matrix, 510 

Boundary conditions, 527 

Boundary value problem (BVP), 445 



Backtracking line search, 346,353 
Backward difference 

approximation, 402 
Backward Euler method 

see Euler' s method of solution 
(implicit form) 
Backward substitution, 156 
Banach fixed-point theorem, 297,422 

see also Contraction theorem 
Banach space, 68 

Basic QR iterations algorithm, 512 
Bauer-Fike theorem, 520 
Bernoulli's differential equation, 426 
Bernoulli's inequality, 123 
Bernoulli numbers, 398 
Bessel's inequality, 217 
"Big O" notation, 155,296 
Binary number codes 

One's complement, 57 

Sign-magnitude, 56 



C, 38,339,565 
C++, 38 
Cancellation, 44 

Cantor's intersection theorem, 294 
Cartesian product, 2 
Catastrophic cancellation, 62,117 
Catastrophic convergence, 90 
Cauchy sequence, 64 
Cauchy-Schwarz inequality, 139 
Cayley-Hamilton theorem, 489 
Central difference approximation, 402 
Chaotic sequence, 323 
Characteristic equation, 458,549 
Characteristic impedance, 538 
Characteristic impedance of free 

space, 539 
Characteristic polynomial, 144,481 
Chebyshev approximation, 239 
Chebyshev norm, 239 
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Chebyshev polynomials of the first kind, 

218-225 
Chebyshev polynomials 

of the second kind, 243-245 
Cholesky decomposition, 167,355 
Christoffel-Darboux formula, 211,389 
Christoffel numbers, 389 
Closed subset, 299 
Cofactor, 489 

Colpitts oscillator, 419,465,538 
Compactly supported, 278 
Companion matrix, 515,519 
Complementary error function, 98 
Complementary sine integral, 100 
Complete solution, 532 
Complete space, 64 
Complete spline, 275 
Completing the square, 131 
Complex numbers 

Cartesian form, 28 

Imaginary part, 31 

Magnitude, 29 

Polar form, 28 

Real part, 31 
Computational complexity, 154 
Computational efficiency, 89 
Computational region, 551 
Condition number of a matrix, 

135-147 
Conic section, 519,526 
Conjugate gradient methods, 527 
Conjugate symmetry, 35 
Continuous mapping, 11 
Contraction (contractive) 

mapping, 184,297 
Contraction theorem, 183,298 
Contra-identity matrix, 199 
Convergence in the mean, 217 
Convergence of a sequence, 63 
Convex function, 352 
Convolution, 204 
Convolution integral, 104,444 
Coordinate rotation digital computing 
(CORDIC) method, 70,107-116 
Countably infinite set, 22 



Courant-Friedrichs-Lewy (CFL) 

condition, 547 
Courant parameter, 546 
Crank-Nicolson method, 528 
Cubic B-spline, 272 
Current-controlled current source 

(CCCS), 420 
Curve fitting, 252 
Cyphertext sequence, 325 



D'Alembert solution, 537 
Daubechies 4-tap scaling function, 520 
Daubechies wavelets, 18 
Deadbeat synchronization, 326 
Defective matrix, 483 
Deflation theorem, 507 
Degree of freedom, 440 
Descent algorithm, 352 
Detection threshold, 333 
Diagonal dominance, 182,281 
Diagonalizable matrix, 445 
Diffusion equation 

see Heat equation 
Dimension, 22 
Dirichlet kernel, 74,103 
Discrete basis, 70,108 
Discrete convolution, 34 
Discrete Fourier series, 25 
Discrete Fourier transform 

(DFT), 27,37,112 
Divergent sequence, 64 
Divided differences, 257-259 
Divided-difference table, 264 
Dominant eigenvalue, 498 
Double sequence, 69 
Duffing equation, 

415,447-448,453-454 



Eavesdropper, 331 
Eigenf unction, 532 
Eigenpair, 480 
Eigenproblem, 480 
Eigenspace, 483 
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Eigenvalue 

Matrix, 143,480 
Partial differential equation 
(PDE), 532 

Eigenvector, 480 

Electric flux density, 535 

Electrical conductivity, 540 

Elementary logic 
Logical contrary, 32 
Logically equivalent, 32 
Necessary condition, 31 
Sufficient condition, 31 

Elliptic integrals, 410-411 

Elliptic partial differential 
equation, 526 

Equality constraint, 357 

Error control coding, 327 

Encryption key, 325 

Energy 

Of a signal (analog), 13 
Of a sequence, 13 

Error 

Absolute, 45,60,140 
Relative, 45,60,140,195 

Euler-Maclaurin formula, 397 

Error function, 94 

Euclidean metric, 6 

Euclidean space, 12,16 

Euler's identity, 19,30 

Exchange matrix 

see Contra-identity matrix 



Faraday's law, 535 

Fast Fourier transform (FFT), 27 

Feasible direction, 357 

Feasible incremental step, 357 

Feasible set, 357 

Fejer's theorem, 122 

Finite difference (FD) method, 545-550 

Finite-difference time-domain (FDTD) 

method, 525,540,550-557 
Finite-element method (FEL), 528 
Finite impulse response (FIR) filtering, 

34,367 



Finite-precision arithmetic effects, 1 
Fixed-point dynamic range, 39 
Fixed-point number 

representations, 38-41 
Fixed-point overflow, 40 
Fixed-point method, 296-305,312-318 
Fixed-point rounding, 41 
Floating-point chopping, 47 
Floating-point dynamic range, 42 
Floating-point exponent, 42 
Floating-point mantissa, 42 
Floating-point normalization, 45 
Floating-point number 

representations, 42-47 
Floating-point overflow, 43 
Floating-point rounding, 43 
Floating-point underflow, 43 
Flop, 154 

Floquet theory, 522 
Forcing term, 548 
FORTRAN, 38 

Forward difference approximation, 401 
Forward elimination, 156 
Forward substitution 

see Forward elimination 
Fourier series, 18,73 
Free space, 535 
Fresnel coefficients, 543 
Frobenius norm, 140 
Full rank matrix, 161 
Function space, 4 
Fundamental mode, 532 
Fundamental theorem 

of algebra, 254,484 



Gamma function, 90 
Gaussian function 

see Gaussian pulse 
Gaussian probability 

density function, 413 
Gaussian pulse, 92,556 
Gauss-Seidel method, 181,527 
Gauss-Seidel successive overrelaxation 

(SOR) method, 181 



TLFeBOOK 



582 



INDEX 



Gauss transformation matrix, 148 

Gauss vector, 149 

Generalized eigenvectors, 485 

Generalized Fourier series 
expansion, 218 

Generalized mean-value theorem, 79 

Generating functions 

Hermite polynomials, 227 
Legendre polynomials, 233 

Geometric series, 34 

Gibbs phenomenon, 77 

Givens matrix, 514 

Global truncation error, 548 

Golden ratio, 347 

Golden ratio search method, 346 

Gradient operator, 342 

Greatest lower bound 
see Infimum 

Group speed, 556 

Haar condition, 239 

Haar scaling function, 17,36 

Haar wavelet, 17,36 

Hankel matrix, 412 

Harmonics, 162,532 

Heat equation, 527 

Henon map, 324 

Hermite' s interpolation formula, 

267-269,387 
Hermite polynomials, 225-229 
Hessenberg matrix, 205,512 
Hessenberg reduction, 513 
Hessian matrix, 344 
Hilbert matrix, 133 
Hilbert space, 68 
Holder inequality, 139 
Homogeneous material, 535 
Householder transformation, 169-170 
Hump phenomenon, 496 
Hyperbolic partial differential 

equation, 526 
Hysteresis, 535 

Ideal diode, 106 
Idempotent matrix, 170 



IEEE floating-point standard, 42 
If and only if (iff), 32 
Ill-conditioned matrix, 132 
Increment function, 437 
Incremental condition estimation 

(ICE), 367 
Inequality 

Cauchy-Schwarz, 15 
Holder, 10 
Minkowski, 10 
Schwarz, 24,36 
Infimum, 5 

Initial conditions, 528-529 
Initial value problem (IVP) 

Adams-Bashforth (AB) methods 

of solution, 459-461 
Adams-Moulton (AM) methods 

of solution, 461-462 
Chaotic instability, 437 
Definition, 415-416 
Euler's method of solution (explicit 

form), 423,443 
Euler method of solution (implicit 

form), 427,443 
Global error, 465 

Heun's method of solution, 433,453 
Midpoint method of solution, 

457-458 
Model problem, 426,443 
Multistep predictor-corrector 

methods of solution, 456-457 
Parasitic term, 459 
Runge-Kutta methods of solution, 

437-441 
Runge-Kutta-Fehlberg (RKF) 

methods of solution, 464-466 
Single-step methods of solution, 455 
Stability, 421,423,427,433,436, 
445-447,458-459,462-463 
Stability regions, 463 
Stiff systems, 432,467-469 
Trapezoidal method of solution, 436 
Truncation error per step, 432-434 
Inner product, 14 
Inner product space, 14 
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Interface between media, 541 
Integral mean-value theorem, 96 
Integration between zeros, 385 
Integration by parts, 90 
Intermediate value theorem, 292,383 
Interpolation 

Hermite, 266-269,381 

Lagrange, 252-257 

Newton, 257-266 

Polynomial, 251 

Spline, 269-284 
Interpolatory graphical display 

algorithm (IGDA), 521 
Invariance (2-norm), 168 
Invariant subspace, 511 
Inverse discrete Fourier transformi 

(IDFT), 27 
Inverse power algorithm, 503 
Isotropic material, 535 



Jacobi algorithm, 203 
Jacobi method, 181,527 
Jacobi overrelaxation 

(JOR) method, 181 
Jacobian matrix, 321 
Jordan blocks, 484 
Jordan canonical form, 483 



Kernel function, 215 

Key size, 324,329 

Kirchoffs laws, 418 

Known plaintext attack, 331,339 

Kronecker delta sequence, 22 



L'Hopital's rule, 81 

Lagrange multipliers, 142,166,357-358 

Lagrange polynomials, 254 

Lagrangian function, 359 

Laplace transform, 416 

Law of cosines, 169 

Leading principle submatrix, 151 

Leapfrog algorithm, 551 



Least upper bound 

see Supremum 
Least-squares, 127-128,132,161 
Least-squares approximation using 

orthogonal polynomials, 235 
Lebesgue integrable functions, 67 
Legendre polynomials, 229-235 
Levinson-Durbin algorithm, 200 
Limit of a sequence, 63 
Line searches, 345-353 
Linearly independent, 21 
Lipschitz condition, 421 
Lipschitz constant, 421 
Lissajous figure, 478 
Local minima, 343 
Local minimizer, 365 
Local truncation error, 545 
Logistic equation, 436 
Logistic map, 323,437 
Lossless material, 539 
Lower triangular matrix, 148 
LU decomposition, 148 



Machine epsilon, 53 

Maclaurin series, 85 

Magnetic conductivity, 540 

Magnetic flux density, 535 

Manchester pulse, 17 

Mathematical induction, 116 

MATLAB, 34,54,90,102,117-118,126, 
129,134-135,137,146,185, 
186-191, 197-198,201,205-206, 
244-246,248,287-289,292,336, 
338-339,351,362-366,409-411, 
413,442,465-467,469-472, 
475-479,498,509,522-524,552, 
557-559,561,563-564,565-577 

Matrix exponential, 444,488-498 

Matrix exponential condition 
number, 497 

Matrix norm, 140 

Matrix norm equivalence, 141 

Matrix operator, 176 

Matrix spectrum, 509 
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Maxwell's equations, 535,540 

Mean value theorem, 79,303 

Message sequence, 325 

Method of false position, 333 

Method of separation of variables, 530 

Metric, 6 

Metric induced by the norm, 1 1 

Metric space, 6 

Micromechanical resonator, 416 

Minor, 489 

Moments, 140,411 



Rectangular rule, 372 

Recursive trapezoidal rule, 398 

Richardson's extrapolation, 396 

Right-point rule, 372 

Romberg integration formula, 397 

Romberg table, 397,399 

Simpson's rule, 378-385 

Simpson's rule truncation error, 
380,383 

Trapezoidal rule, 371-378 

Trapezoidal rule truncation error for- 
mula, 375-376 



Natural spline, 275 

Negative definite matrix, 365 

Nesting property, 199 

Newton's method, 353-356 

Newton's method with backtracking 
line search, 354-356 

Newton-Raphson method breakdown 
phenomena, 311-312 

Newton-Raphson method, 305-312, 
318-323 

Newton-Raphson method rate of con- 
vergence, 309-311 

Nondefective matrix, 483 

Nonexpansive mapping, 297 

Nonlinear observer, 326 

Non-return- to-zero (NRZ) pulse, 17 

Norm, 10 

Normal equations, 164 

Normal matrix, 147,498 

Normal mode, 532 

Normed space, 10 

Numerical dispersion, 550,555-556 

Numerical integration 

Chebyshev-Gauss quadrature, 389 
Composite Simpson's rule, 380 
Composite trapezoidal rule, 376 
Corrected trapezoidal rule, 394 
Gaussian quadrature, 389 
Hermite quadrature, 388 
Left-point rule, 372 
Legendre-Gauss quadrature, 391 
Midpoint rule, 373,409 



Objective function, 341 
Orbit 

see Trajectory 
Order of a partial differential equation, 

525 
Ordinary differential equation 

(ODE), 415 
Orthogonality, 17 
Orthogonal matrix, 161 
Orthogonal polynomial, 208 
Orthonormal set, 23 
Overdetermined least-squares 

approximation, 161 
Overrelaxation method, 182 
Overshoot, 76-77 
Overtones 

see Harmonics 



Parabolic partial differential 

equation, 526 
Parallelogram equality, 15 
Parametric amplifier, 473 
Parseval's equality, 216 
Partial differential equation (PDE), 525 
PASCAL, 38 
Perfectly matched layer (PML), 

540,544,553 
Permeability, 535 
Permittivity, 535 
Permutation matrix, 205 
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Persymmetry property, 199 

Phase portrait, 454 

Phase speed, 555 

Phasor analysis, 539 

Picard's theorem, 422 

Pivot, 151 

Plane electromagnetic waves, 534-535 

Plane waves, 536 

Pointwise convergence, 71 

Poisson equation, 526 

Polynomial, 4 

Positive definite matrix, 130 

Positive semidefinite matrix, 130 

Power method, 498-500 

Power series, 94 

Probability of false alarm, 333 

Product method 

see Method of separation of variables 
Projection operator, 170,197 
Propagation constant, 539 
Pseudoin verse, 192 
Pumping frequency, 473 



QR decomposition, 161 
QR factorization 

see QR decomposition 
QR iterations, 508-518 
Quadratic form, 129,164 
Quantization, 39 
Quantization error, 40 
Quartz crystal, 416 



Radius of convergence, 95 

Rate of convergence, 105,296 

Ratio test, 233 

Rayleigh quotient iteration, 508,523 

Real Schur decomposition, 511 

Reflection coefficient, 543 

Reflection coefficients of a Toeplitz 

matrix, 201 
Regula falsi 

see Method of false position 
Relative permeability, 535 



Relative permittivity, 535 
Relative perturbations, 158 
Residual reflections, 555 
Residual vector, 144-145,168,180 
Riemann integrable functions, 68-69 
Ringing artifacts, 77 
Rodrigues formula, 213,230 
Rolle's theorem, 265 
Rosenbrock's function, 342 
Rotation operator, 112-113,120,484 
Rounding errors, 38 
Roundoff errors 

see Rounding errors 
Runge's phenomenon, 257 



Saddle point, 343 

Scaled power algorithm, 580 

Schelin's theorem, 109 

Second-order linear partial differential 

equation, 526 
Sequence, 3 

Sequence of partial sums, 74,122 
Sherman-Morrison-Woodbury 

formula, 203 
Shift parameter, 502,516 
Similarity transformation, 487 
Simultaneous diagonalizability, 520 
Sine integral, 100 
Single-shift QR iterations 

algorithm, 517 
Singular values, 147,165 
Singular value decomposition 

(SVD), 164 
Singular vectors, 165 
Stability, 157 
State variables, 416 
State vector, 325 
Stationary point, 343 
Stiffness quotient, 479 
Stirling's formula, 91 
Strange attractor, 466 
Strongly nonsingular matrix 

see Strongly regular matrix 
Strongly regular matrix, 201 
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Subharmonics, 162 
Submultiplicative property of matrix 

norms, 141 
Summation by parts, 249 
Spectral norm, 147 
Spectral radius, 177 
Spectrum, 532 
Speed of light, 538 
Supremum, 5 
Symmetric part of a matrix, 203 



Unique global minimizer, 344 

Unitary space, 12,16 

Unit circle, 248,502 

Unit roundoff, 53 

Unit sphere, 141 

Unit step function, 421 

Unit vector, 17,170 

UNIX, 565 

Upper quasi-triangular matrix, 512 

Upper triangular matrix, 148 



Taylor series, 78-96 

Taylor polynomial, 85 

Taylor's theorem, 440 

Three-term recurrence formula (relation) 

Definition, 208 

Chebyshev polynomials of the first 
kind, 222 

Chebyshev polynomials of the second 
kind, 243 

Gram polynomials, 249 

Hermite polynomials, 229 

Legendre polynomials, 234 
Toeplitz matrix, 199 
Trace of a matrix, 206 
Trajectory, 454 
Transmission coefficient, 543 
Transverse electromagnetic (TEM) 

waves, 586 
Tridiagonal matrix, 199,275,516 
Truncation error, 85 
Truncation error per step, 546 
Two-dimensional Taylor series, 

318-319 
Tychonov regularization, 368 



Uncertainty principle, 36 
Underrelaxation method, 182 
Uniform approximation, 222,238 
Uniform convergence, 71 



Vandermonde matrix, 254,285 

Varactor diode, 473 

Vector electric field intensity, 534 

Vector magnetic field intensity, 535 

Vector, 9 

Vector dot product, 16,48 

Vector Hermitian transpose, 16 

Vector norm, 138 

Vector norm equivalence, 139 

Vector space, 9 

Vector subspace, 239 

Vector transpose, 16 

Vibrating string, 528-534 

Von Neumann stability analysis, 548 

Wavelength, 538 
Wavelet series, 18 
Wave equation 

Electric field, 536 

Magnetic field, 536 
Weierstrass' approximation 

theorem, 216 
Weierstrass function, 384 
Weighted inner product, 207 
Weighting function, 207 
Well-conditioned matrix, 137 



Zeroth order modified Bessel function 
of the first kind, 400 
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